From mboxrd@z Thu Jan 1 00:00:00 1970 From: Oded Gabbay Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver Date: Mon, 21 Jul 2014 20:28:50 +0300 Message-ID: <53CD4DD2.10906@amd.com> References: <53C7D645.3070607@amd.com> <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <20140721152511.GW15237@phenom.ffwll.local> <20140721155851.GB4519@gmail.com> <20140721170546.GB15237@phenom.ffwll.local> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from na01-bn1-obe.outbound.protection.outlook.com (dns-bn1lp0143.outbound.protection.outlook.com [207.46.163.143]) by gabe.freedesktop.org (Postfix) with ESMTP id 448566E28D for ; Mon, 21 Jul 2014 10:29:03 -0700 (PDT) In-Reply-To: <20140721170546.GB15237@phenom.ffwll.local> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Jerome Glisse , =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , David Airlie , Alex Deucher , Andrew Morton , John Bridgman , Joerg Roedel , Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , Ben Goz , Alexey Skidanov , Evgeny Pinchuk , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm List-Id: dri-devel@lists.freedesktop.org T24gMjEvMDcvMTQgMjA6MDUsIERhbmllbCBWZXR0ZXIgd3JvdGU6Cj4gT24gTW9uLCBKdWwgMjEs IDIwMTQgYXQgMTE6NTg6NTJBTSAtMDQwMCwgSmVyb21lIEdsaXNzZSB3cm90ZToKPj4gT24gTW9u LCBKdWwgMjEsIDIwMTQgYXQgMDU6MjU6MTFQTSArMDIwMCwgRGFuaWVsIFZldHRlciB3cm90ZToK Pj4+IE9uIE1vbiwgSnVsIDIxLCAyMDE0IGF0IDAzOjM5OjA5UE0gKzAyMDAsIENocmlzdGlhbiBL w7ZuaWcgd3JvdGU6Cj4+Pj4gQW0gMjEuMDcuMjAxNCAxNDozNiwgc2NocmllYiBPZGVkIEdhYmJh eToKPj4+Pj4gT24gMjAvMDcvMTQgMjA6NDYsIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4+Pj4+PiBP biBUaHUsIEp1bCAxNywgMjAxNCBhdCAwNDo1NzoyNVBNICswMzAwLCBPZGVkIEdhYmJheSB3cm90 ZToKPj4+Pj4+PiBGb3Jnb3QgdG8gY2MgbWFpbGluZyBsaXN0IG9uIGNvdmVyIGxldHRlci4gU29y cnkuCj4+Pj4+Pj4KPj4+Pj4+PiBBcyBhIGNvbnRpbnVhdGlvbiB0byB0aGUgZXhpc3RpbmcgZGlz Y3Vzc2lvbiwgaGVyZSBpcyBhIHYyIHBhdGNoIHNlcmllcwo+Pj4+Pj4+IHJlc3RydWN0dXJlZCB3 aXRoIGEgY2xlYW5lciBoaXN0b3J5IGFuZCBubwo+Pj4+Pj4+IHRvdGFsbHktZGlmZmVyZW50LWVh cmx5LXZlcnNpb25zCj4+Pj4+Pj4gb2YgdGhlIGNvZGUuCj4+Pj4+Pj4KPj4+Pj4+PiBJbnN0ZWFk IG9mIDgzIHBhdGNoZXMsIHRoZXJlIGFyZSBub3cgYSB0b3RhbCBvZiAyNSBwYXRjaGVzLCB3aGVy ZSA1IG9mCj4+Pj4+Pj4gdGhlbQo+Pj4+Pj4+IGFyZSBtb2RpZmljYXRpb25zIHRvIHJhZGVvbiBk cml2ZXIgYW5kIDE4IG9mIHRoZW0gaW5jbHVkZSBvbmx5IGFtZGtmZAo+Pj4+Pj4+IGNvZGUuCj4+ Pj4+Pj4gVGhlcmUgaXMgbm8gY29kZSBnb2luZyBhd2F5IG9yIGV2ZW4gbW9kaWZpZWQgYmV0d2Vl biBwYXRjaGVzLCBvbmx5Cj4+Pj4+Pj4gYWRkZWQuCj4+Pj4+Pj4KPj4+Pj4+PiBUaGUgZHJpdmVy IHdhcyByZW5hbWVkIGZyb20gcmFkZW9uX2tmZCB0byBhbWRrZmQgYW5kIG1vdmVkIHRvIHJlc2lk ZQo+Pj4+Pj4+IHVuZGVyCj4+Pj4+Pj4gZHJtL3JhZGVvbi9hbWRrZmQuIFRoaXMgbW92ZSB3YXMg ZG9uZSB0byBlbXBoYXNpemUgdGhlIGZhY3QgdGhhdCB0aGlzCj4+Pj4+Pj4gZHJpdmVyCj4+Pj4+ Pj4gaXMgYW4gQU1ELW9ubHkgZHJpdmVyIGF0IHRoaXMgcG9pbnQuIEhhdmluZyBzYWlkIHRoYXQs IHdlIGRvIGZvcmVzZWUgYQo+Pj4+Pj4+IGdlbmVyaWMgaHNhIGZyYW1ld29yayBiZWluZyBpbXBs ZW1lbnRlZCBpbiB0aGUgZnV0dXJlIGFuZCBpbiB0aGF0Cj4+Pj4+Pj4gY2FzZSwgd2UKPj4+Pj4+ PiB3aWxsIGFkanVzdCBhbWRrZmQgdG8gd29yayB3aXRoaW4gdGhhdCBmcmFtZXdvcmsuCj4+Pj4+ Pj4KPj4+Pj4+PiBBcyB0aGUgYW1ka2ZkIGRyaXZlciBzaG91bGQgc3VwcG9ydCBtdWx0aXBsZSBB TUQgZ2Z4IGRyaXZlcnMsIHdlIHdhbnQKPj4+Pj4+PiB0bwo+Pj4+Pj4+IGtlZXAgaXQgYXMgYSBz ZXBlcmF0ZSBkcml2ZXIgZnJvbSByYWRlb24uIFRoZXJlZm9yZSwgdGhlIGFtZGtmZCBjb2RlIGlz Cj4+Pj4+Pj4gY29udGFpbmVkIGluIGl0cyBvd24gZm9sZGVyLiBUaGUgYW1ka2ZkIGZvbGRlciB3 YXMgcHV0IHVuZGVyIHRoZSByYWRlb24KPj4+Pj4+PiBmb2xkZXIgYmVjYXVzZSB0aGUgb25seSBB TUQgZ2Z4IGRyaXZlciBpbiB0aGUgTGludXgga2VybmVsIGF0IHRoaXMKPj4+Pj4+PiBwb2ludAo+ Pj4+Pj4+IGlzIHRoZSByYWRlb24gZHJpdmVyLiBIYXZpbmcgc2FpZCB0aGF0LCB3ZSB3aWxsIHBy b2JhYmx5IG5lZWQgdG8gbW92ZQo+Pj4+Pj4+IGl0Cj4+Pj4+Pj4gKG1heWJlIHRvIGJlIGRpcmVj dGx5IHVuZGVyIGRybSkgYWZ0ZXIgd2UgaW50ZWdyYXRlIHdpdGggYWRkaXRpb25hbAo+Pj4+Pj4+ IEFNRCBnZngKPj4+Pj4+PiBkcml2ZXJzLgo+Pj4+Pj4+Cj4+Pj4+Pj4gRm9yIHBlb3BsZSB3aG8g bGlrZSB0byByZXZpZXcgdXNpbmcgZ2l0LCB0aGUgdjIgcGF0Y2ggc2V0IGlzIGxvY2F0ZWQKPj4+ Pj4+PiBhdDoKPj4+Pj4+PiBodHRwOi8vY2dpdC5mcmVlZGVza3RvcC5vcmcvfmdhYmJheW8vbGlu dXgvbG9nLz9oPWtmZC1uZXh0LTMuMTctdjIKPj4+Pj4+Pgo+Pj4+Pj4+IFdyaXR0ZW4gYnkgT2Rl ZCBHYWJiYXloIDxvZGVkLmdhYmJheUBhbWQuY29tPgo+Pj4+Pj4KPj4+Pj4+IFNvIHF1aWNrIGNv bW1lbnRzIGJlZm9yZSBpIGZpbmlzaCBnb2luZyBvdmVyIGFsbCBwYXRjaGVzLiBUaGVyZSBpcyBt YW55Cj4+Pj4+PiB0aGluZ3MgdGhhdCBuZWVkIG1vcmUgZG9jdW1lbnRhdGlvbiBlc3BhY2lhbHkg YXMgb2YgcmlnaHQgbm93IHRoZXJlIGlzCj4+Pj4+PiBubyB1c2Vyc3BhY2UgaSBjYW4gZ28gbG9v ayBhdC4KPj4+Pj4gU28gcXVpY2sgY29tbWVudHMgb24gc29tZSBvZiB5b3VyIHF1ZXN0aW9ucyBi dXQgZmlyc3Qgb2YgYWxsLCB0aGFua3MgZm9yCj4+Pj4+IHRoZSB0aW1lIHlvdSBkZWRpY2F0ZWQg dG8gcmV2aWV3IHRoZSBjb2RlLgo+Pj4+Pj4KPj4+Pj4+IFRoZXJlIGZldyBzaG93IHN0b3BwZXIs IGJpZ2dlc3Qgb25lIGlzIGdwdSBtZW1vcnkgcGlubmluZyB0aGlzIGlzIGEgYmlnCj4+Pj4+PiBu bywgdGhhdCB3b3VsZCBuZWVkIHNlcmlvdXMgYXJndW1lbnRzIGZvciBhbnkgaG9wZSBvZiBjb252 aW5jaW5nIG1lIG9uCj4+Pj4+PiB0aGF0IHNpZGUuCj4+Pj4+IFdlIG9ubHkgZG8gZ3B1IG1lbW9y eSBwaW5uaW5nIGZvciBrZXJuZWwgb2JqZWN0cy4gVGhlcmUgYXJlIG5vIHVzZXJzcGFjZQo+Pj4+ PiBvYmplY3RzIHRoYXQgYXJlIHBpbm5lZCBvbiB0aGUgZ3B1IG1lbW9yeSBpbiBvdXIgZHJpdmVy LiBJZiB0aGF0IGlzIHRoZQo+Pj4+PiBjYXNlLCBpcyBpdCBzdGlsbCBhIHNob3cgc3RvcHBlciA/ Cj4+Pj4+Cj4+Pj4+IFRoZSBrZXJuZWwgb2JqZWN0cyBhcmU6Cj4+Pj4+IC0gcGlwZWxpbmVzICg0 IHBlciBkZXZpY2UpCj4+Pj4+IC0gbXFkIHBlciBoaXEgKG9ubHkgMSBwZXIgZGV2aWNlKQo+Pj4+ PiAtIG1xZCBwZXIgdXNlcnNwYWNlIHF1ZXVlLiBPbiBLViwgd2Ugc3VwcG9ydCB1cCB0byAxSyBx dWV1ZXMgcGVyIHByb2Nlc3MsCj4+Pj4+IGZvciBhIHRvdGFsIG9mIDUxMksgcXVldWVzLiBFYWNo IG1xZCBpcyAxNTEgYnl0ZXMsIGJ1dCB0aGUgYWxsb2NhdGlvbiBpcwo+Pj4+PiBkb25lIGluIDI1 NiBhbGlnbm1lbnQuIFNvIHRvdGFsICpwb3NzaWJsZSogbWVtb3J5IGlzIDEyOE1CCj4+Pj4+IC0g a2VybmVsIHF1ZXVlIChvbmx5IDEgcGVyIGRldmljZSkKPj4+Pj4gLSBmZW5jZSBhZGRyZXNzIGZv ciBrZXJuZWwgcXVldWUKPj4+Pj4gLSBydW5saXN0cyBmb3IgdGhlIENQICgxIG9yIDIgcGVyIGRl dmljZSkKPj4+Pgo+Pj4+IFRoZSBtYWluIHF1ZXN0aW9ucyBoZXJlIGFyZSBpZiBpdCdzIGF2b2lk IGFibGUgdG8gcGluIGRvd24gdGhlIG1lbW9yeSBhbmQgaWYKPj4+PiB0aGUgbWVtb3J5IGlzIHBp bm5lZCBkb3duIGF0IGRyaXZlciBsb2FkLCBieSByZXF1ZXN0IGZyb20gdXNlcnNwYWNlIG9yIGJ5 Cj4+Pj4gYW55dGhpbmcgZWxzZS4KPj4+Pgo+Pj4+IEFzIGZhciBhcyBJIGNhbiBzZWUgb25seSB0 aGUgIm1xZCBwZXIgdXNlcnNwYWNlIHF1ZXVlIiBtaWdodCBiZSBhIGJpdAo+Pj4+IHF1ZXN0aW9u YWJsZSwgZXZlcnl0aGluZyBlbHNlIHNvdW5kcyByZWFzb25hYmxlLgo+Pj4KPj4+IEFzaWRlLCBp OTE1IHBlcnNwZWN0aXZlIGFnYWluIChpLmUuIGhvdyB3ZSBzb2x2ZWQgdGhpcyk6IFdoZW4gc2No ZWR1bGluZwo+Pj4gYXdheSBmcm9tIGNvbnRleHRzIHdlIHVucGluIHRoZW0gYW5kIHB1dCB0aGVt IGludG8gdGhlIGxydS4gQW5kIGluIHRoZQo+Pj4gc2hyaW5rZXIgd2UgaGF2ZSBhIGxhc3QtZGl0 Y2ggY2FsbGJhY2sgdG8gc3dpdGNoIHRvIGEgZGVmYXVsdCBjb250ZXh0Cj4+PiAoc2luY2UgeW91 IGNhbid0IGV2ZXIgaGF2ZSBubyBjb250ZXh0IG9uY2UgeW91J3ZlIHN0YXJ0ZWQpIHdoaWNoIG1l YW5zIHdlCj4+PiBjYW4gZXZpY3QgYW55IGNvbnRleHQgb2JqZWN0IGlmIGl0J3MgZ2V0dGluZyBp biB0aGUgd2F5Lgo+Pgo+PiBTbyBJbnRlbCBoYXJkd2FyZSByZXBvcnQgdGhyb3VnaCBzb21lIGlu dGVycnVwdCBvciBzb21lIGNoYW5uZWwgd2hlbiBpdCBpcwo+PiBub3QgdXNpbmcgYSBjb250ZXh0 ID8gaWUga2VybmVsIHNpZGUgZ2V0IG5vdGlmaWNhdGlvbiB3aGVuIHNvbWUgdXNlciBjb250ZXh0 Cj4+IGlzIGRvbmUgZXhlY3V0aW5nID8KPiAKPiBZZXMsIGFzIGxvbmcgYXMgd2UgZG8gdGhlIHNj aGVkdWxpbmcgd2l0aCB0aGUgY3B1IHdlIGdldCBpbnRlcnJ1cHRzIGZvcgo+IGNvbnRleHQgc3dp dGNoZXMuIFRoZSBtZWNoYW5pYyBpcyBhbHJlYWR5IHB1Ymxpc2hlZCBpbiB0aGUgZXhlY2xpc3QK PiBwYXRjaGVzIGN1cnJlbnRseSBmbG9hdGluZyBhcm91bmQuIFdlIGdldCBhIHNwZWNpYWwgY29u dGV4dCBzd2l0Y2gKPiBpbnRlcnJ1cHQuCj4gCj4gQnV0IHdlIGhhdmUgdGhpcyB1bnBpbiBsb2dp YyBhbHJlYWR5IG9uIHRoZSBjdXJyZW50IGNvZGUgd2hlcmUgd2Ugc3dpdGNoCj4gY29udGV4dHMg dGhyb3VnaCBpbi1saW5lIGNzIGNvbW1hbmRzIGZyb20gdGhlIGtlcm5lbC4gVGhlcmUgd2Ugb2J2 aW91c2x5Cj4gdXNlIHRoZSBub3JtYWwgYmF0Y2ggY29tcGxldGlvbiBldmVudHMuCj4gCj4+IFRo ZSBpc3N1ZSB3aXRoIHJhZGVvbiBoYXJkd2FyZSBBRkFJQ1QgaXMgdGhhdCB0aGUgaGFyZHdhcmUg ZG8gbm90IHJlcG9ydCBhbnkKPj4gdGhpbmcgYWJvdXQgdGhlIHVzZXJzcGFjZSBjb250ZXh0IHJ1 bm5pbmcgaWUgeW91IGRvIG5vdCBnZXQgbm90aWZpY2F0aW9uIHdoZW4KPj4gYSBjb250ZXh0IGlz IG5vdCB1c2UuIFdlbGwgQUZBSUNULiBNYXliZSBoYXJkd2FyZSBkbyBwcm92aWRlIHRoYXQuCj4g Cj4gSSdtIG5vdCBzdXJlIHdoZXRoZXIgd2UgY2FuIGRvIHRoZSBzYW1lIHRyaWNrIHdpdGggdGhl IGh3IHNjaGVkdWxlci4gQnV0Cj4gdGhlbiB1bnBpbm5pbmcgaHcgY29udGV4dHMgd2lsbCBkcmFp biB0aGUgcGlwZWxpbmUgYW55d2F5LCBzbyBJIGd1ZXNzIHdlCj4gY2FuIGp1c3Qgc3RvcCBmZWVk aW5nIHRoZSBodyBzY2hlZHVsZXIgdW50aWwgaXQgcnVucyBkcnkuIEFuZCB0aGVuIHVucGluCj4g YW5kIGV2aWN0LgpTbywgSSdtIGFmcmFpZCBidXQgd2UgY2FuJ3QgZG8gdGhpcyBmb3IgQU1EIEth dmVyaSBiZWNhdXNlOgoKYS4gVGhlIGh3IHNjaGVkdWxlciBkb2Vzbid0IGluZm9ybSB1cyB3aGlj aCBxdWV1ZXMgaXQgaXMgZ29pbmcgdG8KZXhlY3V0ZSBuZXh0LiBXZSBmZWVkIGl0IGEgcnVubGlz dCBvZiBxdWV1ZXMsIHdoaWNoIGNhbiBiZSB2ZXJ5IGxhcmdlCih3ZSBoYXZlIGEgdGVzdCB0aGF0 IHJ1bnMgMTAwMCBxdWV1ZXMgb24gdGhlIHNhbWUgcnVubGlzdCwgYnV0IHdlIGNhbgpwdXQgYSBs b3QgbW9yZSkuIEFsbCB0aGUgTVFEcyBvZiB0aG9zZSBxdWV1ZXMgbXVzdCBiZSBwaW5uZWQgaW4g bWVtb3J5CmFzIGxvbmcgYXMgdGhlIHJ1bmxpc3QgaXMgaW4gZWZmZWN0LiBUaGUgcnVubGlzdCBp cyBpbiBlZmZlY3QgdW50aWwKZWl0aGVyIGEgcXVldWUgaXMgZGVsZXRlZCBvciBhIHF1ZXVlIGlz IGFkZGVkIChvciBzb21ldGhpbmcgbW9yZSBleHRyZW1lCmhhcHBlbnMsIGxpa2UgdGhlIHByb2Nl c3MgdGVybWluYXRlcykuCgpiLiBUaGUgaHcgc2NoZWR1bGVyIHRha2VzIGNhcmUgb2YgVk1JRCB0 byBQQVNJRCBtYXBwaW5nLiBXZSBkb24ndApwcm9ncmFtIHRoZSBBVEMgcmVnaXN0ZXJzIG1hbnVh bGx5LCB0aGUgaW50ZXJuYWwgQ1AgZG9lcyB0aGF0CmR5bmFtaWNhbGx5LCBzbyB3ZSBiYXNpY2Fs bHkgaGF2ZSBvdmVyLXN1YnNjcmlwdGlvbiBvZiBwcm9jZXNzZXMgYXMKd2VsbC4gVGhlcmVmb3Jl LCB3ZSBjYW4ndCBwaW5nIE1RRHMgYmFzZWQgb24gVk1JRCBiaW5kaW5nLgoKSSBkb24ndCBzZWUg QU1EIG1vdmluZyBiYWNrIHRvIFNXIHNjaGVkdWxpbmcsIGFzIGl0IGRvZXNuJ3Qgc2NhbGUgd2Vs bAp3aXRoIHRoZSBudW1iZXIgb2YgcHJvY2Vzc2VzIGFuZCBxdWV1ZXMgYW5kIG91ciBuZXh0IGdl biBBUFUgd2lsbCBoYXZlIGEKbG90IG1vcmUgcXVldWVzIHRoYW4gd2hhdCB3ZSBoYXZlIG9uIEtW CgoJT2RlZAo+IAo+PiBMaWtlIHRoZSBWTUlEIGlzIGEgbGltaXRlZCByZXNvdXJjZXMgc28geW91 IGhhdmUgdG8gZHluYW1pY2x5IGJpbmQgdGhlbSBzbwo+PiBtYXliZSB3ZSBjYW4gb25seSBhbGxv Y2F0ZSBwaW5uZWQgYnVmZmVyIGZvciBlYWNoIFZNSUQgYW5kIHRoZW4gd2hlbiBiaW5kaW5nCj4+ IGEgUEFTSUQgdG8gYSBWTUlEIGl0IGFsc28gY29weSBiYWNrIHBpbm5lZCBidWZmZXIgdG8gcGFz aWQgdW5waW5uZWQgY29weS4KPiAKPiBZZWFoLCBwYXNpZCBhc3NpZ25tZW50IHdpbGwgYmUgZnVu LiBOb3Qgc3VyZSB3aGV0aGVyIEplc3NlJ3MgcGF0Y2hlcyB3aWxsCj4gZG8gdGhpcyBhbHJlYWR5 LiBXZSBfZG9fIGFscmVhZHkgaGF2ZSBmdW4gd2l0aCBjdHggaWQgYXNzaWdtZW50cyB0aG91Z2gK PiBzaW5jZSB3ZSBtb3ZlIHRoZW0gYXJvdW5kIChhbmQgdGhlIGh3IGlkIGlzIHRoZSBnZ3R0IGFk ZHJlc3MgYWZhaWspLiBTbyB3ZQo+IG5lZWQgdG8gcmVtYXAgdGhlbSBhbHJlYWR5LiBOb3Qgc3Vy ZSBvbiB0aGUgZGV0YWlscyBmb3IgcGFzaWQgbWFwcGluZywKPiBpaXJjIGl0J3MgYSBzZXBhcmF0 ZSBmaWVsZCBzb21ld2hlcmUgaW4gdGhlIGNvbnRleHQgc3RydWN0LiBKZXNzZSBrbm93cwo+IHRo ZSBkZXRhaWxzLgo+IC1EYW5pZWwKPiAKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fCmRyaS1kZXZlbCBtYWlsaW5nIGxpc3QKZHJpLWRldmVsQGxpc3RzLmZy ZWVkZXNrdG9wLm9yZwpodHRwOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGlu Zm8vZHJpLWRldmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f181.google.com (mail-pd0-f181.google.com [209.85.192.181]) by kanga.kvack.org (Postfix) with ESMTP id 2B3866B006C for ; Mon, 21 Jul 2014 13:29:11 -0400 (EDT) Received: by mail-pd0-f181.google.com with SMTP id g10so7965996pdj.26 for ; Mon, 21 Jul 2014 10:29:10 -0700 (PDT) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0208.outbound.protection.outlook.com. [207.46.163.208]) by mx.google.com with ESMTPS id gp1si13055470pbd.145.2014.07.21.10.29.09 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 21 Jul 2014 10:29:10 -0700 (PDT) Message-ID: <53CD4DD2.10906@amd.com> Date: Mon, 21 Jul 2014 20:28:50 +0300 From: Oded Gabbay MIME-Version: 1.0 Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver References: <53C7D645.3070607@amd.com> <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <20140721152511.GW15237@phenom.ffwll.local> <20140721155851.GB4519@gmail.com> <20140721170546.GB15237@phenom.ffwll.local> In-Reply-To: <20140721170546.GB15237@phenom.ffwll.local> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Jerome Glisse , =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , David Airlie , Alex Deucher , Andrew Morton , John Bridgman , Joerg Roedel , Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , Ben Goz , Alexey Skidanov , Evgeny Pinchuk , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm On 21/07/14 20:05, Daniel Vetter wrote: > On Mon, Jul 21, 2014 at 11:58:52AM -0400, Jerome Glisse wrote: >> On Mon, Jul 21, 2014 at 05:25:11PM +0200, Daniel Vetter wrote: >>> On Mon, Jul 21, 2014 at 03:39:09PM +0200, Christian K=C3=B6nig wrote: >>>> Am 21.07.2014 14:36, schrieb Oded Gabbay: >>>>> On 20/07/14 20:46, Jerome Glisse wrote: >>>>>> On Thu, Jul 17, 2014 at 04:57:25PM +0300, Oded Gabbay wrote: >>>>>>> Forgot to cc mailing list on cover letter. Sorry. >>>>>>> >>>>>>> As a continuation to the existing discussion, here is a v2 patch = series >>>>>>> restructured with a cleaner history and no >>>>>>> totally-different-early-versions >>>>>>> of the code. >>>>>>> >>>>>>> Instead of 83 patches, there are now a total of 25 patches, where= 5 of >>>>>>> them >>>>>>> are modifications to radeon driver and 18 of them include only am= dkfd >>>>>>> code. >>>>>>> There is no code going away or even modified between patches, onl= y >>>>>>> added. >>>>>>> >>>>>>> The driver was renamed from radeon_kfd to amdkfd and moved to res= ide >>>>>>> under >>>>>>> drm/radeon/amdkfd. This move was done to emphasize the fact that = this >>>>>>> driver >>>>>>> is an AMD-only driver at this point. Having said that, we do fore= see a >>>>>>> generic hsa framework being implemented in the future and in that >>>>>>> case, we >>>>>>> will adjust amdkfd to work within that framework. >>>>>>> >>>>>>> As the amdkfd driver should support multiple AMD gfx drivers, we = want >>>>>>> to >>>>>>> keep it as a seperate driver from radeon. Therefore, the amdkfd c= ode is >>>>>>> contained in its own folder. The amdkfd folder was put under the = radeon >>>>>>> folder because the only AMD gfx driver in the Linux kernel at thi= s >>>>>>> point >>>>>>> is the radeon driver. Having said that, we will probably need to = move >>>>>>> it >>>>>>> (maybe to be directly under drm) after we integrate with addition= al >>>>>>> AMD gfx >>>>>>> drivers. >>>>>>> >>>>>>> For people who like to review using git, the v2 patch set is loca= ted >>>>>>> at: >>>>>>> http://cgit.freedesktop.org/~gabbayo/linux/log/?h=3Dkfd-next-3.17= -v2 >>>>>>> >>>>>>> Written by Oded Gabbayh >>>>>> >>>>>> So quick comments before i finish going over all patches. There is= many >>>>>> things that need more documentation espacialy as of right now ther= e is >>>>>> no userspace i can go look at. >>>>> So quick comments on some of your questions but first of all, thank= s for >>>>> the time you dedicated to review the code. >>>>>> >>>>>> There few show stopper, biggest one is gpu memory pinning this is = a big >>>>>> no, that would need serious arguments for any hope of convincing m= e on >>>>>> that side. >>>>> We only do gpu memory pinning for kernel objects. There are no user= space >>>>> objects that are pinned on the gpu memory in our driver. If that is= the >>>>> case, is it still a show stopper ? >>>>> >>>>> The kernel objects are: >>>>> - pipelines (4 per device) >>>>> - mqd per hiq (only 1 per device) >>>>> - mqd per userspace queue. On KV, we support up to 1K queues per pr= ocess, >>>>> for a total of 512K queues. Each mqd is 151 bytes, but the allocati= on is >>>>> done in 256 alignment. So total *possible* memory is 128MB >>>>> - kernel queue (only 1 per device) >>>>> - fence address for kernel queue >>>>> - runlists for the CP (1 or 2 per device) >>>> >>>> The main questions here are if it's avoid able to pin down the memor= y and if >>>> the memory is pinned down at driver load, by request from userspace = or by >>>> anything else. >>>> >>>> As far as I can see only the "mqd per userspace queue" might be a bi= t >>>> questionable, everything else sounds reasonable. >>> >>> Aside, i915 perspective again (i.e. how we solved this): When schedul= ing >>> away from contexts we unpin them and put them into the lru. And in th= e >>> shrinker we have a last-ditch callback to switch to a default context >>> (since you can't ever have no context once you've started) which mean= s we >>> can evict any context object if it's getting in the way. >> >> So Intel hardware report through some interrupt or some channel when i= t is >> not using a context ? ie kernel side get notification when some user c= ontext >> is done executing ? >=20 > Yes, as long as we do the scheduling with the cpu we get interrupts for > context switches. The mechanic is already published in the execlist > patches currently floating around. We get a special context switch > interrupt. >=20 > But we have this unpin logic already on the current code where we switc= h > contexts through in-line cs commands from the kernel. There we obviousl= y > use the normal batch completion events. >=20 >> The issue with radeon hardware AFAICT is that the hardware do not repo= rt any >> thing about the userspace context running ie you do not get notificati= on when >> a context is not use. Well AFAICT. Maybe hardware do provide that. >=20 > I'm not sure whether we can do the same trick with the hw scheduler. Bu= t > then unpinning hw contexts will drain the pipeline anyway, so I guess w= e > can just stop feeding the hw scheduler until it runs dry. And then unpi= n > and evict. So, I'm afraid but we can't do this for AMD Kaveri because: a. The hw scheduler doesn't inform us which queues it is going to execute next. We feed it a runlist of queues, which can be very large (we have a test that runs 1000 queues on the same runlist, but we can put a lot more). All the MQDs of those queues must be pinned in memory as long as the runlist is in effect. The runlist is in effect until either a queue is deleted or a queue is added (or something more extreme happens, like the process terminates). b. The hw scheduler takes care of VMID to PASID mapping. We don't program the ATC registers manually, the internal CP does that dynamically, so we basically have over-subscription of processes as well. Therefore, we can't ping MQDs based on VMID binding. I don't see AMD moving back to SW scheduling, as it doesn't scale well with the number of processes and queues and our next gen APU will have a lot more queues than what we have on KV Oded >=20 >> Like the VMID is a limited resources so you have to dynamicly bind the= m so >> maybe we can only allocate pinned buffer for each VMID and then when b= inding >> a PASID to a VMID it also copy back pinned buffer to pasid unpinned co= py. >=20 > Yeah, pasid assignment will be fun. Not sure whether Jesse's patches wi= ll > do this already. We _do_ already have fun with ctx id assigments though > since we move them around (and the hw id is the ggtt address afaik). So= we > need to remap them already. Not sure on the details for pasid mapping, > iirc it's a separate field somewhere in the context struct. Jesse knows > the details. > -Daniel >=20 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933300AbaGUR3R (ORCPT ); Mon, 21 Jul 2014 13:29:17 -0400 Received: from mail-bl2lp0210.outbound.protection.outlook.com ([207.46.163.210]:37920 "EHLO na01-bl2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932320AbaGUR3P convert rfc822-to-8bit (ORCPT ); Mon, 21 Jul 2014 13:29:15 -0400 X-WSS-ID: 0N92OK6-08-4C3-02 X-M-MSG: Message-ID: <53CD4DD2.10906@amd.com> Date: Mon, 21 Jul 2014 20:28:50 +0300 From: Oded Gabbay Organization: AMD User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Jerome Glisse , =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , David Airlie , Alex Deucher , Andrew Morton , "John Bridgman" , Joerg Roedel , "Andrew Lewycky" , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , Ben Goz , Alexey Skidanov , Evgeny Pinchuk , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver References: <53C7D645.3070607@amd.com> <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <20140721152511.GW15237@phenom.ffwll.local> <20140721155851.GB4519@gmail.com> <20140721170546.GB15237@phenom.ffwll.local> In-Reply-To: <20140721170546.GB15237@phenom.ffwll.local> Content-Type: text/plain; charset="UTF-8" X-Originating-IP: [10.224.155.153] Content-Transfer-Encoding: 8BIT X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:165.204.84.222;CTRY:US;IPV:NLI;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(6009001)(428002)(24454002)(199002)(189002)(479174003)(51704005)(51404002)(52314003)(65956001)(81342001)(87936001)(47776003)(74502001)(76176999)(85306003)(21056001)(101416001)(33656002)(79102001)(20776003)(83072002)(92566001)(2201001)(97736001)(65806001)(92726001)(50986999)(54356999)(64706001)(83322001)(86362001)(107886001)(93886003)(36756003)(81542001)(77982001)(85852003)(15202345003)(64126003)(44976005)(76482001)(80022001)(4396001)(106466001)(15975445006)(68736004)(31966008)(95666004)(46102001)(102836001)(84676001)(19580405001)(65816999)(83506001)(99396002)(74662001)(23676002)(19580395003)(50466002)(107046002)(105586002)(1121002)(921003);DIR:OUT;SFP:;SCL:1;SRVR:BY2PR02MB041;H:atltwp02.amd.com;FPR:;MLV:sfv;PTR:InfoDomainNonexistent;MX:1;LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0279B3DD0D Authentication-Results: spf=none (sender IP is 165.204.84.222) smtp.mailfrom=Oded.Gabbay@amd.com; X-OriginatorOrg: amd4.onmicrosoft.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 21/07/14 20:05, Daniel Vetter wrote: > On Mon, Jul 21, 2014 at 11:58:52AM -0400, Jerome Glisse wrote: >> On Mon, Jul 21, 2014 at 05:25:11PM +0200, Daniel Vetter wrote: >>> On Mon, Jul 21, 2014 at 03:39:09PM +0200, Christian König wrote: >>>> Am 21.07.2014 14:36, schrieb Oded Gabbay: >>>>> On 20/07/14 20:46, Jerome Glisse wrote: >>>>>> On Thu, Jul 17, 2014 at 04:57:25PM +0300, Oded Gabbay wrote: >>>>>>> Forgot to cc mailing list on cover letter. Sorry. >>>>>>> >>>>>>> As a continuation to the existing discussion, here is a v2 patch series >>>>>>> restructured with a cleaner history and no >>>>>>> totally-different-early-versions >>>>>>> of the code. >>>>>>> >>>>>>> Instead of 83 patches, there are now a total of 25 patches, where 5 of >>>>>>> them >>>>>>> are modifications to radeon driver and 18 of them include only amdkfd >>>>>>> code. >>>>>>> There is no code going away or even modified between patches, only >>>>>>> added. >>>>>>> >>>>>>> The driver was renamed from radeon_kfd to amdkfd and moved to reside >>>>>>> under >>>>>>> drm/radeon/amdkfd. This move was done to emphasize the fact that this >>>>>>> driver >>>>>>> is an AMD-only driver at this point. Having said that, we do foresee a >>>>>>> generic hsa framework being implemented in the future and in that >>>>>>> case, we >>>>>>> will adjust amdkfd to work within that framework. >>>>>>> >>>>>>> As the amdkfd driver should support multiple AMD gfx drivers, we want >>>>>>> to >>>>>>> keep it as a seperate driver from radeon. Therefore, the amdkfd code is >>>>>>> contained in its own folder. The amdkfd folder was put under the radeon >>>>>>> folder because the only AMD gfx driver in the Linux kernel at this >>>>>>> point >>>>>>> is the radeon driver. Having said that, we will probably need to move >>>>>>> it >>>>>>> (maybe to be directly under drm) after we integrate with additional >>>>>>> AMD gfx >>>>>>> drivers. >>>>>>> >>>>>>> For people who like to review using git, the v2 patch set is located >>>>>>> at: >>>>>>> http://cgit.freedesktop.org/~gabbayo/linux/log/?h=kfd-next-3.17-v2 >>>>>>> >>>>>>> Written by Oded Gabbayh >>>>>> >>>>>> So quick comments before i finish going over all patches. There is many >>>>>> things that need more documentation espacialy as of right now there is >>>>>> no userspace i can go look at. >>>>> So quick comments on some of your questions but first of all, thanks for >>>>> the time you dedicated to review the code. >>>>>> >>>>>> There few show stopper, biggest one is gpu memory pinning this is a big >>>>>> no, that would need serious arguments for any hope of convincing me on >>>>>> that side. >>>>> We only do gpu memory pinning for kernel objects. There are no userspace >>>>> objects that are pinned on the gpu memory in our driver. If that is the >>>>> case, is it still a show stopper ? >>>>> >>>>> The kernel objects are: >>>>> - pipelines (4 per device) >>>>> - mqd per hiq (only 1 per device) >>>>> - mqd per userspace queue. On KV, we support up to 1K queues per process, >>>>> for a total of 512K queues. Each mqd is 151 bytes, but the allocation is >>>>> done in 256 alignment. So total *possible* memory is 128MB >>>>> - kernel queue (only 1 per device) >>>>> - fence address for kernel queue >>>>> - runlists for the CP (1 or 2 per device) >>>> >>>> The main questions here are if it's avoid able to pin down the memory and if >>>> the memory is pinned down at driver load, by request from userspace or by >>>> anything else. >>>> >>>> As far as I can see only the "mqd per userspace queue" might be a bit >>>> questionable, everything else sounds reasonable. >>> >>> Aside, i915 perspective again (i.e. how we solved this): When scheduling >>> away from contexts we unpin them and put them into the lru. And in the >>> shrinker we have a last-ditch callback to switch to a default context >>> (since you can't ever have no context once you've started) which means we >>> can evict any context object if it's getting in the way. >> >> So Intel hardware report through some interrupt or some channel when it is >> not using a context ? ie kernel side get notification when some user context >> is done executing ? > > Yes, as long as we do the scheduling with the cpu we get interrupts for > context switches. The mechanic is already published in the execlist > patches currently floating around. We get a special context switch > interrupt. > > But we have this unpin logic already on the current code where we switch > contexts through in-line cs commands from the kernel. There we obviously > use the normal batch completion events. > >> The issue with radeon hardware AFAICT is that the hardware do not report any >> thing about the userspace context running ie you do not get notification when >> a context is not use. Well AFAICT. Maybe hardware do provide that. > > I'm not sure whether we can do the same trick with the hw scheduler. But > then unpinning hw contexts will drain the pipeline anyway, so I guess we > can just stop feeding the hw scheduler until it runs dry. And then unpin > and evict. So, I'm afraid but we can't do this for AMD Kaveri because: a. The hw scheduler doesn't inform us which queues it is going to execute next. We feed it a runlist of queues, which can be very large (we have a test that runs 1000 queues on the same runlist, but we can put a lot more). All the MQDs of those queues must be pinned in memory as long as the runlist is in effect. The runlist is in effect until either a queue is deleted or a queue is added (or something more extreme happens, like the process terminates). b. The hw scheduler takes care of VMID to PASID mapping. We don't program the ATC registers manually, the internal CP does that dynamically, so we basically have over-subscription of processes as well. Therefore, we can't ping MQDs based on VMID binding. I don't see AMD moving back to SW scheduling, as it doesn't scale well with the number of processes and queues and our next gen APU will have a lot more queues than what we have on KV Oded > >> Like the VMID is a limited resources so you have to dynamicly bind them so >> maybe we can only allocate pinned buffer for each VMID and then when binding >> a PASID to a VMID it also copy back pinned buffer to pasid unpinned copy. > > Yeah, pasid assignment will be fun. Not sure whether Jesse's patches will > do this already. We _do_ already have fun with ctx id assigments though > since we move them around (and the hw id is the ggtt address afaik). So we > need to remap them already. Not sure on the details for pasid mapping, > iirc it's a separate field somewhere in the context struct. Jesse knows > the details. > -Daniel >