From mboxrd@z Thu Jan 1 00:00:00 1970 From: Oded Gabbay Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver Date: Mon, 21 Jul 2014 20:42:58 +0300 Message-ID: <53CD5122.5040804@amd.com> References: <53C7D645.3070607@amd.com> <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <53CD1FB6.1000602@amd.com> <20140721155437.GA4519@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1lp0141.outbound.protection.outlook.com [207.46.163.141]) by gabe.freedesktop.org (Postfix) with ESMTP id A8C386E003 for ; Mon, 21 Jul 2014 10:43:11 -0700 (PDT) In-Reply-To: <20140721155437.GA4519@gmail.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Jerome Glisse Cc: Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm , Evgeny Pinchuk , Alexey Skidanov , Andrew Morton List-Id: dri-devel@lists.freedesktop.org T24gMjEvMDcvMTQgMTg6NTQsIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4gT24gTW9uLCBKdWwgMjEs IDIwMTQgYXQgMDU6MTI6MDZQTSArMDMwMCwgT2RlZCBHYWJiYXkgd3JvdGU6Cj4+IE9uIDIxLzA3 LzE0IDE2OjM5LCBDaHJpc3RpYW4gS8O2bmlnIHdyb3RlOgo+Pj4gQW0gMjEuMDcuMjAxNCAxNDoz Niwgc2NocmllYiBPZGVkIEdhYmJheToKPj4+PiBPbiAyMC8wNy8xNCAyMDo0NiwgSmVyb21lIEds aXNzZSB3cm90ZToKPj4+Pj4gT24gVGh1LCBKdWwgMTcsIDIwMTQgYXQgMDQ6NTc6MjVQTSArMDMw MCwgT2RlZCBHYWJiYXkgd3JvdGU6Cj4+Pj4+PiBGb3Jnb3QgdG8gY2MgbWFpbGluZyBsaXN0IG9u IGNvdmVyIGxldHRlci4gU29ycnkuCj4+Pj4+Pgo+Pj4+Pj4gQXMgYSBjb250aW51YXRpb24gdG8g dGhlIGV4aXN0aW5nIGRpc2N1c3Npb24sIGhlcmUgaXMgYSB2MiBwYXRjaCBzZXJpZXMKPj4+Pj4+ IHJlc3RydWN0dXJlZCB3aXRoIGEgY2xlYW5lciBoaXN0b3J5IGFuZCBubyB0b3RhbGx5LWRpZmZl cmVudC1lYXJseS12ZXJzaW9ucwo+Pj4+Pj4gb2YgdGhlIGNvZGUuCj4+Pj4+Pgo+Pj4+Pj4gSW5z dGVhZCBvZiA4MyBwYXRjaGVzLCB0aGVyZSBhcmUgbm93IGEgdG90YWwgb2YgMjUgcGF0Y2hlcywg d2hlcmUgNSBvZiB0aGVtCj4+Pj4+PiBhcmUgbW9kaWZpY2F0aW9ucyB0byByYWRlb24gZHJpdmVy IGFuZCAxOCBvZiB0aGVtIGluY2x1ZGUgb25seSBhbWRrZmQgY29kZS4KPj4+Pj4+IFRoZXJlIGlz IG5vIGNvZGUgZ29pbmcgYXdheSBvciBldmVuIG1vZGlmaWVkIGJldHdlZW4gcGF0Y2hlcywgb25s eSBhZGRlZC4KPj4+Pj4+Cj4+Pj4+PiBUaGUgZHJpdmVyIHdhcyByZW5hbWVkIGZyb20gcmFkZW9u X2tmZCB0byBhbWRrZmQgYW5kIG1vdmVkIHRvIHJlc2lkZSB1bmRlcgo+Pj4+Pj4gZHJtL3JhZGVv bi9hbWRrZmQuIFRoaXMgbW92ZSB3YXMgZG9uZSB0byBlbXBoYXNpemUgdGhlIGZhY3QgdGhhdCB0 aGlzIGRyaXZlcgo+Pj4+Pj4gaXMgYW4gQU1ELW9ubHkgZHJpdmVyIGF0IHRoaXMgcG9pbnQuIEhh dmluZyBzYWlkIHRoYXQsIHdlIGRvIGZvcmVzZWUgYQo+Pj4+Pj4gZ2VuZXJpYyBoc2EgZnJhbWV3 b3JrIGJlaW5nIGltcGxlbWVudGVkIGluIHRoZSBmdXR1cmUgYW5kIGluIHRoYXQgY2FzZSwgd2UK Pj4+Pj4+IHdpbGwgYWRqdXN0IGFtZGtmZCB0byB3b3JrIHdpdGhpbiB0aGF0IGZyYW1ld29yay4K Pj4+Pj4+Cj4+Pj4+PiBBcyB0aGUgYW1ka2ZkIGRyaXZlciBzaG91bGQgc3VwcG9ydCBtdWx0aXBs ZSBBTUQgZ2Z4IGRyaXZlcnMsIHdlIHdhbnQgdG8KPj4+Pj4+IGtlZXAgaXQgYXMgYSBzZXBlcmF0 ZSBkcml2ZXIgZnJvbSByYWRlb24uIFRoZXJlZm9yZSwgdGhlIGFtZGtmZCBjb2RlIGlzCj4+Pj4+ PiBjb250YWluZWQgaW4gaXRzIG93biBmb2xkZXIuIFRoZSBhbWRrZmQgZm9sZGVyIHdhcyBwdXQg dW5kZXIgdGhlIHJhZGVvbgo+Pj4+Pj4gZm9sZGVyIGJlY2F1c2UgdGhlIG9ubHkgQU1EIGdmeCBk cml2ZXIgaW4gdGhlIExpbnV4IGtlcm5lbCBhdCB0aGlzIHBvaW50Cj4+Pj4+PiBpcyB0aGUgcmFk ZW9uIGRyaXZlci4gSGF2aW5nIHNhaWQgdGhhdCwgd2Ugd2lsbCBwcm9iYWJseSBuZWVkIHRvIG1v dmUgaXQKPj4+Pj4+IChtYXliZSB0byBiZSBkaXJlY3RseSB1bmRlciBkcm0pIGFmdGVyIHdlIGlu dGVncmF0ZSB3aXRoIGFkZGl0aW9uYWwgQU1EIGdmeAo+Pj4+Pj4gZHJpdmVycy4KPj4+Pj4+Cj4+ Pj4+PiBGb3IgcGVvcGxlIHdobyBsaWtlIHRvIHJldmlldyB1c2luZyBnaXQsIHRoZSB2MiBwYXRj aCBzZXQgaXMgbG9jYXRlZCBhdDoKPj4+Pj4+IGh0dHA6Ly9jZ2l0LmZyZWVkZXNrdG9wLm9yZy9+ Z2FiYmF5by9saW51eC9sb2cvP2g9a2ZkLW5leHQtMy4xNy12Mgo+Pj4+Pj4KPj4+Pj4+IFdyaXR0 ZW4gYnkgT2RlZCBHYWJiYXloIDxvZGVkLmdhYmJheUBhbWQuY29tPgo+Pj4+Pgo+Pj4+PiBTbyBx dWljayBjb21tZW50cyBiZWZvcmUgaSBmaW5pc2ggZ29pbmcgb3ZlciBhbGwgcGF0Y2hlcy4gVGhl cmUgaXMgbWFueQo+Pj4+PiB0aGluZ3MgdGhhdCBuZWVkIG1vcmUgZG9jdW1lbnRhdGlvbiBlc3Bh Y2lhbHkgYXMgb2YgcmlnaHQgbm93IHRoZXJlIGlzCj4+Pj4+IG5vIHVzZXJzcGFjZSBpIGNhbiBn byBsb29rIGF0Lgo+Pj4+IFNvIHF1aWNrIGNvbW1lbnRzIG9uIHNvbWUgb2YgeW91ciBxdWVzdGlv bnMgYnV0IGZpcnN0IG9mIGFsbCwgdGhhbmtzIGZvciB0aGUKPj4+PiB0aW1lIHlvdSBkZWRpY2F0 ZWQgdG8gcmV2aWV3IHRoZSBjb2RlLgo+Pj4+Pgo+Pj4+PiBUaGVyZSBmZXcgc2hvdyBzdG9wcGVy LCBiaWdnZXN0IG9uZSBpcyBncHUgbWVtb3J5IHBpbm5pbmcgdGhpcyBpcyBhIGJpZwo+Pj4+PiBu bywgdGhhdCB3b3VsZCBuZWVkIHNlcmlvdXMgYXJndW1lbnRzIGZvciBhbnkgaG9wZSBvZiBjb252 aW5jaW5nIG1lIG9uCj4+Pj4+IHRoYXQgc2lkZS4KPj4+PiBXZSBvbmx5IGRvIGdwdSBtZW1vcnkg cGlubmluZyBmb3Iga2VybmVsIG9iamVjdHMuIFRoZXJlIGFyZSBubyB1c2Vyc3BhY2UKPj4+PiBv YmplY3RzIHRoYXQgYXJlIHBpbm5lZCBvbiB0aGUgZ3B1IG1lbW9yeSBpbiBvdXIgZHJpdmVyLiBJ ZiB0aGF0IGlzIHRoZSBjYXNlLAo+Pj4+IGlzIGl0IHN0aWxsIGEgc2hvdyBzdG9wcGVyID8KPj4+ Pgo+Pj4+IFRoZSBrZXJuZWwgb2JqZWN0cyBhcmU6Cj4+Pj4gLSBwaXBlbGluZXMgKDQgcGVyIGRl dmljZSkKPj4+PiAtIG1xZCBwZXIgaGlxIChvbmx5IDEgcGVyIGRldmljZSkKPj4+PiAtIG1xZCBw ZXIgdXNlcnNwYWNlIHF1ZXVlLiBPbiBLViwgd2Ugc3VwcG9ydCB1cCB0byAxSyBxdWV1ZXMgcGVy IHByb2Nlc3MsIGZvcgo+Pj4+IGEgdG90YWwgb2YgNTEySyBxdWV1ZXMuIEVhY2ggbXFkIGlzIDE1 MSBieXRlcywgYnV0IHRoZSBhbGxvY2F0aW9uIGlzIGRvbmUgaW4KPj4+PiAyNTYgYWxpZ25tZW50 LiBTbyB0b3RhbCAqcG9zc2libGUqIG1lbW9yeSBpcyAxMjhNQgo+Pj4+IC0ga2VybmVsIHF1ZXVl IChvbmx5IDEgcGVyIGRldmljZSkKPj4+PiAtIGZlbmNlIGFkZHJlc3MgZm9yIGtlcm5lbCBxdWV1 ZQo+Pj4+IC0gcnVubGlzdHMgZm9yIHRoZSBDUCAoMSBvciAyIHBlciBkZXZpY2UpCj4+Pgo+Pj4g VGhlIG1haW4gcXVlc3Rpb25zIGhlcmUgYXJlIGlmIGl0J3MgYXZvaWQgYWJsZSB0byBwaW4gZG93 biB0aGUgbWVtb3J5IGFuZCBpZiB0aGUKPj4+IG1lbW9yeSBpcyBwaW5uZWQgZG93biBhdCBkcml2 ZXIgbG9hZCwgYnkgcmVxdWVzdCBmcm9tIHVzZXJzcGFjZSBvciBieSBhbnl0aGluZwo+Pj4gZWxz ZS4KPj4+Cj4+PiBBcyBmYXIgYXMgSSBjYW4gc2VlIG9ubHkgdGhlICJtcWQgcGVyIHVzZXJzcGFj ZSBxdWV1ZSIgbWlnaHQgYmUgYSBiaXQKPj4+IHF1ZXN0aW9uYWJsZSwgZXZlcnl0aGluZyBlbHNl IHNvdW5kcyByZWFzb25hYmxlLgo+Pj4KPj4+IENocmlzdGlhbi4KPj4KPj4gTW9zdCBvZiB0aGUg cGluIGRvd25zIGFyZSBkb25lIG9uIGRldmljZSBpbml0aWFsaXphdGlvbi4KPj4gVGhlICJtcWQg cGVyIHVzZXJzcGFjZSIgaXMgZG9uZSBwZXIgdXNlcnNwYWNlIHF1ZXVlIGNyZWF0aW9uLiBIb3dl dmVyLCBhcyBJCj4+IHNhaWQsIGl0IGhhcyBhbiB1cHBlciBsaW1pdCBvZiAxMjhNQiBvbiBLViwg YW5kIGNvbnNpZGVyaW5nIHRoZSAyRyBsb2NhbAo+PiBtZW1vcnksIEkgdGhpbmsgaXQgaXMgT0su Cj4+IFRoZSBydW5saXN0cyBhcmUgYWxzbyBkb25lIG9uIHVzZXJzcGFjZSBxdWV1ZSBjcmVhdGlv bi9kZWxldGlvbiwgYnV0IHdlIG9ubHkKPj4gaGF2ZSAxIG9yIDIgcnVubGlzdHMgcGVyIGRldmlj ZSwgc28gaXQgaXMgbm90IHRoYXQgYmFkLgo+IAo+IDJHIGxvY2FsIG1lbW9yeSA/IFlvdSBjYW4g bm90IGFzc3VtZSBhbnl0aGluZyBvbiB1c2Vyc2lkZSBjb25maWd1cmF0aW9uIHNvbWUKPiBvbmUg bWlnaHQgYnVpbGQgYW4gaHNhIGNvbXB1dGVyIHdpdGggNTEyTSBhbmQgc3RpbGwgZXhwZWN0IGEg ZnVuY3Rpb25pbmcKPiBkZXNrdG9wLgpGaXJzdCBvZiBhbGwsIEknbSBvbmx5IGNvbnNpZGVyaW5n IEthdmVyaSBjb21wdXRlciwgbm90ICJoc2EiIGNvbXB1dGVyLgpTZWNvbmQsIEkgd291bGQgaW1h Z2luZSB3ZSBjYW4gYnVpbGQgc29tZSBwcm90ZWN0aW9uIGFyb3VuZCBpdCwgbGlrZQpjaGVja2lu ZyB0b3RhbCBsb2NhbCBtZW1vcnkgYW5kIGxpbWl0IG51bWJlciBvZiBxdWV1ZXMgYmFzZWQgb24g c29tZQpwZXJjZW50YWdlIG9mIHRoYXQgdG90YWwgbG9jYWwgbWVtb3J5LiBTbywgaWYgc29tZW9u ZSB3aWxsIGhhdmUgb25seQo1MTJNLCBoZSB3aWxsIGJlIGFibGUgdG8gb3BlbiBsZXNzIHF1ZXVl cy4KCgo+IAo+IEkgbmVlZCB0byBnbyBsb29rIGludG8gd2hhdCBhbGwgdGhpcyBtcWQgaXMgZm9y LCB3aGF0IGl0IGRvZXMgYW5kIHdoYXQgaXQgaXMKPiBhYm91dC4gQnV0IHBpbm5pbmcgaXMgcmVh bGx5IGJhZCBhbmQgdGhpcyBpcyBhbiBpc3N1ZSB3aXRoIHVzZXJzcGFjZSBjb21tYW5kCj4gc2No ZWR1bGluZyBhbiBpc3N1ZSB0aGF0IG9idmlvdXNseSBBTUQgZmFpbHMgdG8gdGFrZSBpbnRvIGFj Y291bnQgaW4gZGVzaWduCj4gcGhhc2UuCk1heWJlLCBidXQgdGhhdCBpcyB0aGUgSC9XIGRlc2ln biBub24tdGhlLWxlc3MuIFdlIGNhbid0IHZlcnkgd2VsbApjaGFuZ2UgdGhlIEgvVy4KCU9kZWQK PiAKPj4gCU9kZWQKPj4+Cj4+Pj4+Cj4+Pj4+IEl0IG1pZ2h0IGJlIGJldHRlciB0byBhZGQgYSBk cml2ZXJzL2dwdS9kcm0vYW1kIGRpcmVjdG9yeSBhbmQgYWRkIGNvbW1vbgo+Pj4+PiBzdHVmZiB0 aGVyZS4KPj4+Pj4KPj4+Pj4gR2l2ZW4gdGhhdCB0aGlzIGlzIG5vdCBpbnRlbmRlZCB0byBiZSBm aW5hbCBIU0EgYXBpIEFGQUlDVCB0aGVuIGkgd291bGQKPj4+Pj4gc2F5IHRoaXMgZmFyIGJldHRl ciB0byBhdm9pZCB0aGUgd2hvbGUga2ZkIG1vZHVsZSBhbmQgYWRkIGlvY3RsIHRvIHJhZGVvbi4K Pj4+Pj4gVGhpcyB3b3VsZCBhdm9pZCBjcmF6eSBjb21tdW5pY2F0aW9uIGJ0dyByYWRlb24gYW5k IGtmZC4KPj4+Pj4KPj4+Pj4gVGhlIHdob2xlIGFwZXJ0dXJlIGJ1c2luZXNzIG5lZWRzIHNvbWUg c2VyaW91cyBleHBsYW5hdGlvbi4gRXNwZWNpYWx5IGFzCj4+Pj4+IHlvdSB3YW50IHRvIHVzZSB1 c2Vyc3BhY2UgYWRkcmVzcyB0aGVyZSBpcyBub3RoaW5nIHRvIHByZXZlbnQgdXNlcnNwYWNlCj4+ Pj4+IHByb2dyYW0gZnJvbSBhbGxvY2F0aW5nIHRoaW5ncyBhdCBhZGRyZXNzIHlvdSByZXNlcnZl IGZvciBsZHMsIHNjcmF0Y2gsCj4+Pj4+IC4uLiBvbmx5IHNhbmUgd2F5IHdvdWxkIGJlIHRvIG1v dmUgdGhvc2UgbGRzLCBzY3JhdGNoIGluc2lkZSB0aGUgdmlydHVhbAo+Pj4+PiBhZGRyZXNzIHJl c2VydmVkIGZvciBrZXJuZWwgKHNlZSBrZXJuZWwgbWVtb3J5IG1hcCkuCj4+Pj4+Cj4+Pj4+IFRo ZSB3aG9sZSBidXNpbmVzcyBvZiBsb2NraW5nIHBlcmZvcm1hbmNlIGNvdW50ZXIgZm9yIGV4Y2x1 c2l2ZSBwZXIgcHJvY2Vzcwo+Pj4+PiBhY2Nlc3MgaXMgYSBiaWcgTk8uIFdoaWNoIGxlYWRzIG1l IHRvIHRoZSBxdWVzdGlvbmFibGUgdXNlZnVsbG5lc3Mgb2YgdXNlcgo+Pj4+PiBzcGFjZSBjb21t YW5kIHJpbmcuCj4+Pj4gVGhhdCdzIGxpa2Ugc2F5aW5nOiAiV2hpY2ggbGVhZHMgbWUgdG8gdGhl IHF1ZXN0aW9uYWJsZSB1c2VmdWxuZXNzIG9mIEhTQSIuIEkKPj4+PiBmaW5kIGl0IGFuYWxvZ291 cyB0byBhIHNpdHVhdGlvbiB3aGVyZSBhIG5ldHdvcmsgbWFpbnRhaW5lciBuYWNraW5nIGEgZHJp dmVyCj4+Pj4gZm9yIGEgbmV0d29yayBjYXJkLCB3aGljaCBpcyBzbG93ZXIgdGhhbiBhIGRpZmZl cmVudCBuZXR3b3JrIGNhcmQuIERvZXNuJ3QKPj4+PiBzZWVtIHJlYXNvbmFibGUgdGhpcyBzaXR1 YXRpb24gaXMgd291bGQgaGFwcGVuLiBIZSB3b3VsZCBzdGlsbCBwdXQgYm90aCB0aGUKPj4+PiBk cml2ZXJzIGluIHRoZSBrZXJuZWwgYmVjYXVzZSBwZW9wbGUgd2FudCB0byB1c2UgdGhlIEgvVyBh bmQgaXRzIGZlYXR1cmVzLiBTbywKPj4+PiBJIGRvbid0IHRoaW5rIHRoaXMgaXMgYSB2YWxpZCBy ZWFzb24gdG8gTkFDSyB0aGUgZHJpdmVyLgo+IAo+IExldCBtZSByZXBocmFzZSwgZHJvcCB0aGUg dGhlIHBlcmZvcm1hbmNlIGNvdW50ZXIgaW9jdGwgYW5kIG1vZHVsbyBtZW1vcnkgcGlubmluZwo+ IGkgc2VlIG5vIG9iamVjdGlvbi4gSW4gb3RoZXIgd29yZCwgaSBhbSBub3QgTkFDS0lORyB3aG9s ZSBwYXRjaHNldCBpIGFtIE5BQ0tJTkcKPiB0aGUgcGVyZm9ybWFuY2UgaW9jdGwuCj4gCj4gQWdh aW4gdGhpcyBpcyBhbm90aGVyIGFyZ3VtZW50IGZvciByb3VuZCB0cmlwIHRvIHRoZSBrZXJuZWwu IEFzIGluc2lkZSBrZXJuZWwgeW91Cj4gY291bGQgcHJvcGVybHkgZG8gZXhjbHVzaXZlIGdwdSBj b3VudGVyIGFjY2VzcyBhY2Nyb3NzIHNpbmdsZSB1c2VyIGNtZCBidWZmZXIKPiBleGVjdXRpb24u Cj4gCj4+Pj4KPj4+Pj4gSSBvbmx5IHNlZSBpc3N1ZXMgd2l0aCB0aGF0LiBGaXJzdCBhbmQgZm9y ZW1vc3QgaSB3b3VsZAo+Pj4+PiBuZWVkIHRvIHNlZSBzb2xpZCBmaWd1cmVzIHRoYXQga2VybmVs IGlvY3RsIG9yIHN5c2NhbGwgaGFzIGEgaGlnaGVyIGFuCj4+Pj4+IG92ZXJoZWFkIHRoYXQgaXMg bWVhc3VyYWJsZSBpbiBhbnkgbWVhbmluZyBmdWxsIHdheSBhZ2FpbnN0IGEgc2ltcGxlCj4+Pj4+ IGZ1bmN0aW9uIGNhbGwuIEkga25vdyB0aGUgdXNlcnNwYWNlIGNvbW1hbmQgcmluZyBpcyBhIGJp ZyBtYXJrZXRpbmcgZmVhdHVyZXMKPj4+Pj4gdGhhdCBwbGVhc2UgaWdub3JhbnQgdXNlcnNwYWNl IHByb2dyYW1tZXIuIEJ1dCByZWFsbHkgdGhpcyBvbmx5IGJyaW5ncyBpc3N1ZXMKPj4+Pj4gYW5k IGZvciBhYnNvbHV0ZWx5IG5vdCB1cHNpZGUgYWZhaWN0Lgo+Pj4+IFJlYWxseSA/IFlvdSB0aGlu ayB0aGF0IGRvaW5nIGEgY29udGV4dCBzd2l0Y2ggdG8ga2VybmVsIHNwYWNlLCB3aXRoIGFsbCBp dHMKPj4+PiBvdmVyaGVhZCwgaXMgX25vdF8gbW9yZSBleHBhbnNpdmUgdGhhbiBqdXN0IGNhbGxp bmcgYSBmdW5jdGlvbiBpbiB1c2Vyc3BhY2UKPj4+PiB3aGljaCBvbmx5IHB1dHMgYSBidWZmZXIg b24gYSByaW5nIGFuZCB3cml0ZXMgYSBkb29yYmVsbCA/Cj4gCj4gSSBhbSBzYXlpbmcgdGhlIG92 ZXJoZWFkIGlzIG5vdCB0aGF0IGJpZyBhbmQgaXQgcHJvYmFibHkgd2lsbCBub3QgbWF0dGVyIGlu IG1vc3QKPiB1c2VjYXNlLiBGb3IgaW5zdGFuY2UgaSBkaWQgd3JvdGUgdGhlIG1vc3QgdXNlbGVz cyBrZXJuZWwgbW9kdWxlIHRoYXQgYWRkIHR3bwo+IG51bWJlciB0aHJvdWdoIGFuIGlvY3RsICho dHRwOi8vcGVvcGxlLmZyZWVkZXNrdG9wLm9yZy9+Z2xpc3NlL2FkZGVyLnRhcikgYW5kCj4gaXQg dGFrZXMgfjAuMzVtaWNyb3NlY29uZHMgd2l0aCBpb2N0bCB3aGlsZSBmdW5jdGlvbiBpcyB+MC4w MjVtaWNyb3NlY29uZHMgc28KPiBpb2N0bCBpcyAxMyB0aW1lcyBzbG93ZXIuCj4gCj4gTm93IGlm IHRoZXJlIGlzIGVub3VnaCBkYXRhIHRoYXQgc2hvd3MgdGhhdCBhIHNpZ25pZmljYW50IHBlcmNl bnRhZ2Ugb2Ygam9icwo+IHN1Ym1pdGVkIHRvIHRoZSBHUFUgd2lsbCB0YWtlIGxlc3MgdGhhdCAw LjM1bWljcm9zZWNvbmQgdGhlbiB5ZXMgdXNlcnNwYWNlCj4gc2NoZWR1bGluZyBkb2VzIG1ha2Ug c2Vuc2UuIEJ1dCBzbyBmYXIgYWxsIHdlIGhhdmUgaXMgaGFuZHdhdmluZyB3aXRoIG5vIGRhdGEK PiB0byBzdXBwb3J0IGFueSBmYWN0cy4KPiAKPiAKPiBOb3cgaWYgd2Ugd2FudCB0byBzY2hlZHVs ZSBmcm9tIHVzZXJzcGFjZSB0aGFuIHlvdSB3aWxsIG5lZWQgdG8gZG8gc29tZXRoaW5nCj4gYWJv dXQgdGhlIHBpbm5pbmcsIHNvbWV0aGluZyB0aGF0IGdpdmVzIGNvbnRyb2wgdG8ga2VybmVsIHNv IHRoYXQga2VybmVsIGNhbgo+IHVucGluIHdoZW4gaXQgd2FudHMgYW5kIG1vdmUgb2JqZWN0IHdo ZW4gaXQgd2FudHMgbm8gbWF0dGVyIHdoYXQgdXNlcnNwYWNlIGlzCj4gZG9pbmcuCj4gCj4+Pj4+ Cj4+Pj4+IFNvIGkgd291bGQgcmF0aGVyIHNlZSBhIHZlcnkgc2ltcGxlIGlvY3RsIHRoYXQgd3Jp dGUgdGhlIGRvb3JiZWxsIGFuZCBtaWdodAo+Pj4+PiBkbyBtb3JlIHRoYW4gdGhhdCBpbiBjYXNl IG9mIHJpbmcvcXVldWUgb3ZlcmNvbW1pdCB3aGVyZSBpdCB3b3VsZCBmaXJzdCBoYXZlCj4+Pj4+ IHRvIHdhaXQgZm9yIGEgZnJlZSByaW5nL3F1ZXVlIHRvIHNjaGVkdWxlIHN0dWZmLiBUaGlzIHdv dWxkIGFsc28gYWxsb3cgc2FuZQo+Pj4+PiBpbXBsZW1lbnRhdGlvbiBvZiB0aGluZ3MgbGlrZSBw ZXJmb3JtYW5jZSBjb3VudGVyIHRoYXQgY291bGQgYmUgYWNxdWlyZSBieQo+Pj4+PiBrZXJuZWwg Zm9yIGR1cmF0aW9uIG9mIGEgam9iIHN1Ym1pdHRlZCBieSB1c2Vyc3BhY2UuIFdoaWxlIHN0aWxs IG5vdCBvcHRpbWFsCj4+Pj4+IHRoaXMgd291bGQgYmUgYmV0dGVyIHRoYXQgdXNlcnNwYWNlIGxv Y2tpbmcuCj4+Pj4+Cj4+Pj4+Cj4+Pj4+IEkgbWlnaHQgaGF2ZSBtb3JlIHRob3VnaHRzIG9uY2Ug aSBhbSBkb25lIHdpdGggYWxsIHRoZSBwYXRjaGVzLgo+Pj4+Pgo+Pj4+PiBDaGVlcnMsCj4+Pj4+ IErDqXLDtG1lCj4+Pj4+Cj4+Pj4+Pgo+Pj4+Pj4gT3JpZ2luYWwgQ292ZXIgTGV0dGVyOgo+Pj4+ Pj4KPj4+Pj4+IFRoaXMgcGF0Y2ggc2V0IGltcGxlbWVudHMgYSBIZXRlcm9nZW5lb3VzIFN5c3Rl bSBBcmNoaXRlY3R1cmUgKEhTQSkgZHJpdmVyCj4+Pj4+PiBmb3IgcmFkZW9uLWZhbWlseSBHUFVz Lgo+Pj4+Pj4gSFNBIGFsbG93cyBkaWZmZXJlbnQgcHJvY2Vzc29yIHR5cGVzIChDUFVzLCBEU1Bz LCBHUFVzLCBldGMuLikgdG8gc2hhcmUKPj4+Pj4+IHN5c3RlbSByZXNvdXJjZXMgbW9yZSBlZmZl Y3RpdmVseSB2aWEgSFcgZmVhdHVyZXMgaW5jbHVkaW5nIHNoYXJlZCBwYWdlYWJsZQo+Pj4+Pj4g bWVtb3J5LCB1c2Vyc3BhY2UtYWNjZXNzaWJsZSB3b3JrIHF1ZXVlcywgYW5kIHBsYXRmb3JtLWxl dmVsIGF0b21pY3MuIEluCj4+Pj4+PiBhZGRpdGlvbiB0byB0aGUgbWVtb3J5IHByb3RlY3Rpb24g bWVjaGFuaXNtcyBpbiBHUFVWTSBhbmQgSU9NTVV2MiwgdGhlIFNlYQo+Pj4+Pj4gSXNsYW5kcyBm YW1pbHkgb2YgR1BVcyBhbHNvIHBlcmZvcm1zIEhXLWxldmVsIHZhbGlkYXRpb24gb2YgY29tbWFu ZHMgcGFzc2VkCj4+Pj4+PiBpbiB0aHJvdWdoIHRoZSBxdWV1ZXMgKGFrYSByaW5ncykuCj4+Pj4+ Pgo+Pj4+Pj4gVGhlIGNvZGUgaW4gdGhpcyBwYXRjaCBzZXQgaXMgaW50ZW5kZWQgdG8gc2VydmUg Ym90aCBhcyBhIHNhbXBsZSBkcml2ZXIgZm9yCj4+Pj4+PiBvdGhlciBIU0EtY29tcGF0aWJsZSBo YXJkd2FyZSBkZXZpY2VzIGFuZCBhcyBhIHByb2R1Y3Rpb24gZHJpdmVyIGZvcgo+Pj4+Pj4gcmFk ZW9uLWZhbWlseSBwcm9jZXNzb3JzLiBUaGUgY29kZSBpcyBhcmNoaXRlY3RlZCB0byBzdXBwb3J0 IG11bHRpcGxlIENQVXMKPj4+Pj4+IGVhY2ggd2l0aCBjb25uZWN0ZWQgR1BVcywgYWx0aG91Z2gg dGhlIGN1cnJlbnQgaW1wbGVtZW50YXRpb24gZm9jdXNlcyBvbiBhCj4+Pj4+PiBzaW5nbGUgS2F2 ZXJpL0JlcmxpbiBBUFUsIGFuZCB3b3JrcyBhbG9uZ3NpZGUgdGhlIGV4aXN0aW5nIHJhZGVvbiBr ZXJuZWwKPj4+Pj4+IGdyYXBoaWNzIGRyaXZlciAoa2dkKS4KPj4+Pj4+IEFNRCBHUFVzIGRlc2ln bmVkIGZvciB1c2Ugd2l0aCBIU0EgKFNlYSBJc2xhbmRzIGFuZCB1cCkgc2hhcmUgc29tZSBoYXJk d2FyZQo+Pj4+Pj4gZnVuY3Rpb25hbGl0eSBiZXR3ZWVuIEhTQSBjb21wdXRlIGFuZCByZWd1bGFy IGdmeC9jb21wdXRlIChtZW1vcnksCj4+Pj4+PiBpbnRlcnJ1cHRzLCByZWdpc3RlcnMpLCB3aGls ZSBvdGhlciBmdW5jdGlvbmFsaXR5IGhhcyBiZWVuIGFkZGVkCj4+Pj4+PiBzcGVjaWZpY2FsbHkg Zm9yIEhTQSBjb21wdXRlICAoaHcgc2NoZWR1bGVyIGZvciB2aXJ0dWFsaXplZCBjb21wdXRlIHJp bmdzKS4KPj4+Pj4+IEFsbCBzaGFyZWQgaGFyZHdhcmUgaXMgb3duZWQgYnkgdGhlIHJhZGVvbiBn cmFwaGljcyBkcml2ZXIsIGFuZCBhbiBpbnRlcmZhY2UKPj4+Pj4+IGJldHdlZW4ga2ZkIGFuZCBr Z2QgYWxsb3dzIHRoZSBrZmQgdG8gbWFrZSB1c2Ugb2YgdGhvc2Ugc2hhcmVkIHJlc291cmNlcywK Pj4+Pj4+IHdoaWxlIEhTQS1zcGVjaWZpYyBmdW5jdGlvbmFsaXR5IGlzIG1hbmFnZWQgZGlyZWN0 bHkgYnkga2ZkIGJ5IHN1Ym1pdHRpbmcKPj4+Pj4+IHBhY2tldHMgaW50byBhbiBIU0Etc3BlY2lm aWMgY29tbWFuZCBxdWV1ZSAodGhlICJISVEiKS4KPj4+Pj4+Cj4+Pj4+PiBEdXJpbmcga2ZkIG1v ZHVsZSBpbml0aWFsaXphdGlvbiBhIGNoYXIgZGV2aWNlIG5vZGUgKC9kZXYva2ZkKSBpcyBjcmVh dGVkCj4+Pj4+PiAoc3Vydml2aW5nIHVudGlsIG1vZHVsZSBleGl0KSwgd2l0aCBpb2N0bHMgZm9y IHF1ZXVlIGNyZWF0aW9uICYgbWFuYWdlbWVudCwKPj4+Pj4+IGFuZCBkYXRhIHN0cnVjdHVyZXMg YXJlIGluaXRpYWxpemVkIGZvciBtYW5hZ2luZyBIU0EgZGV2aWNlIHRvcG9sb2d5Lgo+Pj4+Pj4g VGhlIHJlc3Qgb2YgdGhlIGluaXRpYWxpemF0aW9uIGlzIGRyaXZlbiBieSBjYWxscyBmcm9tIHRo ZSByYWRlb24ga2dkIGF0IHRoZQo+Pj4+Pj4gZm9sbG93aW5nIHBvaW50cyA6Cj4+Pj4+Pgo+Pj4+ Pj4gLSByYWRlb25faW5pdCAoa2ZkX2luaXQpCj4+Pj4+PiAtIHJhZGVvbl9leGl0IChrZmRfZmlu aSkKPj4+Pj4+IC0gcmFkZW9uX2RyaXZlcl9sb2FkX2ttcyAoa2ZkX2RldmljZV9wcm9iZSwga2Zk X2RldmljZV9pbml0KQo+Pj4+Pj4gLSByYWRlb25fZHJpdmVyX3VubG9hZF9rbXMgKGtmZF9kZXZp Y2VfZmluaSkKPj4+Pj4+Cj4+Pj4+PiBEdXJpbmcgdGhlIHByb2JlIGFuZCBpbml0IHByb2Nlc3Np bmcgcGVyLWRldmljZSBkYXRhIHN0cnVjdHVyZXMgYXJlCj4+Pj4+PiBlc3RhYmxpc2hlZCB3aGlj aCBjb25uZWN0IHRvIHRoZSBhc3NvY2lhdGVkIGdyYXBoaWNzIGtlcm5lbCBkcml2ZXIuIFRoaXMK Pj4+Pj4+IGluZm9ybWF0aW9uIGlzIGV4cG9zZWQgdG8gdXNlcnNwYWNlIHZpYSBzeXNmcywgYWxv bmcgd2l0aCBhIHZlcnNpb24gbnVtYmVyCj4+Pj4+PiBhbGxvd2luZyB1c2Vyc3BhY2UgdG8gZGV0 ZXJtaW5lIGlmIGEgdG9wb2xvZ3kgY2hhbmdlIGhhcyBvY2N1cnJlZCB3aGlsZSBpdAo+Pj4+Pj4g d2FzIHJlYWRpbmcgZnJvbSBzeXNmcy4KPj4+Pj4+IFRoZSBpbnRlcmZhY2UgYmV0d2VlbiBrZmQg YW5kIGtnZCBhbHNvIGFsbG93cyB0aGUga2ZkIHRvIHJlcXVlc3QgYnVmZmVyCj4+Pj4+PiBtYW5h Z2VtZW50IHNlcnZpY2VzIGZyb20ga2dkLCBhbmQgYWxsb3dzIGtnZCB0byByb3V0ZSBpbnRlcnJ1 cHQgcmVxdWVzdHMgdG8KPj4+Pj4+IGtmZCBjb2RlIHNpbmNlIHRoZSBpbnRlcnJ1cHQgYmxvY2sg aXMgc2hhcmVkIGJldHdlZW4gcmVndWxhcgo+Pj4+Pj4gZ3JhcGhpY3MvY29tcHV0ZSBhbmQgSFNB IGNvbXB1dGUgc3Vic3lzdGVtcyBpbiB0aGUgR1BVLgo+Pj4+Pj4KPj4+Pj4+IFRoZSBrZmQgY29k ZSB3b3JrcyB3aXRoIGFuIG9wZW4gc291cmNlIHVzZXJtb2RlIGxpYnJhcnkgKCJsaWJoc2FrbXQi KSB3aGljaAo+Pj4+Pj4gaXMgaW4gdGhlIGZpbmFsIHN0YWdlcyBvZiBJUCByZXZpZXcgYW5kIHNo b3VsZCBiZSBwdWJsaXNoZWQgaW4gYSBzZXBhcmF0ZQo+Pj4+Pj4gcmVwbyBvdmVyIHRoZSBuZXh0 IGZldyBkYXlzLgo+Pj4+Pj4gVGhlIGNvZGUgb3BlcmF0ZXMgaW4gb25lIG9mIHRocmVlIG1vZGVz LCBzZWxlY3RhYmxlIHZpYSB0aGUgc2NoZWRfcG9saWN5Cj4+Pj4+PiBtb2R1bGUgcGFyYW1ldGVy IDoKPj4+Pj4+Cj4+Pj4+PiAtIHNjaGVkX3BvbGljeT0wIHVzZXMgYSBoYXJkd2FyZSBzY2hlZHVs ZXIgcnVubmluZyBpbiB0aGUgTUVDIGJsb2NrIHdpdGhpbgo+Pj4+Pj4gQ1AsIGFuZCBhbGxvd3Mg b3ZlcnN1YnNjcmlwdGlvbiAobW9yZSBxdWV1ZXMgdGhhbiBIVyBzbG90cykKPj4+Pj4+IC0gc2No ZWRfcG9saWN5PTEgYWxzbyB1c2VzIEhXIHNjaGVkdWxpbmcgYnV0IGRvZXMgbm90IGFsbG93Cj4+ Pj4+PiBvdmVyc3Vic2NyaXB0aW9uLCBzbyBjcmVhdGVfcXVldWUgcmVxdWVzdHMgZmFpbCB3aGVu IHdlIHJ1biBvdXQgb2YgSFcgc2xvdHMKPj4+Pj4+IC0gc2NoZWRfcG9saWN5PTIgZG9lcyBub3Qg dXNlIEhXIHNjaGVkdWxpbmcsIHNvIHRoZSBkcml2ZXIgbWFudWFsbHkgYXNzaWducwo+Pj4+Pj4g cXVldWVzIHRvIEhXIHNsb3RzIGJ5IHByb2dyYW1taW5nIHJlZ2lzdGVycwo+Pj4+Pj4KPj4+Pj4+ IFRoZSAibm8gSFcgc2NoZWR1bGluZyIgb3B0aW9uIGlzIGZvciBkZWJ1ZyAmIG5ldyBoYXJkd2Fy ZSBicmluZ3VwIG9ubHksIHNvCj4+Pj4+PiBoYXMgbGVzcyB0ZXN0IGNvdmVyYWdlIHRoYW4gdGhl IG90aGVyIG9wdGlvbnMuIERlZmF1bHQgaW4gdGhlIGN1cnJlbnQgY29kZQo+Pj4+Pj4gaXMgIkhX IHNjaGVkdWxpbmcgd2l0aG91dCBvdmVyc3Vic2NyaXB0aW9uIiBzaW5jZSB0aGF0IGlzIHdoZXJl IHdlIGhhdmUgdGhlCj4+Pj4+PiBtb3N0IHRlc3QgY292ZXJhZ2UgYnV0IHdlIGV4cGVjdCB0byBj aGFuZ2UgdGhlIGRlZmF1bHQgdG8gIkhXIHNjaGVkdWxpbmcKPj4+Pj4+IHdpdGggb3ZlcnN1YnNj cmlwdGlvbiIgYWZ0ZXIgZnVydGhlciB0ZXN0aW5nLiBUaGlzIGVmZmVjdGl2ZWx5IHJlbW92ZXMg dGhlCj4+Pj4+PiBIVyBsaW1pdCBvbiB0aGUgbnVtYmVyIG9mIHdvcmsgcXVldWVzIGF2YWlsYWJs ZSB0byBhcHBsaWNhdGlvbnMuCj4+Pj4+Pgo+Pj4+Pj4gUHJvZ3JhbXMgcnVubmluZyBvbiB0aGUg R1BVIGFyZSBhc3NvY2lhdGVkIHdpdGggYW4gYWRkcmVzcyBzcGFjZSB0aHJvdWdoIHRoZQo+Pj4+ Pj4gVk1JRCBmaWVsZCwgd2hpY2ggaXMgdHJhbnNsYXRlZCB0byBhIHVuaXF1ZSBQQVNJRCBhdCBh Y2Nlc3MgdGltZSB2aWEgYSBzZXQKPj4+Pj4+IG9mIDE2IFZNSUQtdG8tUEFTSUQgbWFwcGluZyBy ZWdpc3RlcnMuIFRoZSBhdmFpbGFibGUgVk1JRHMgKGN1cnJlbnRseSAxNikKPj4+Pj4+IGFyZSBw YXJ0aXRpb25lZCAodW5kZXIgY29udHJvbCBvZiB0aGUgcmFkZW9uIGtnZCkgYmV0d2VlbiBjdXJy ZW50Cj4+Pj4+PiBnZngvY29tcHV0ZSBhbmQgSFNBIGNvbXB1dGUsIHdpdGggZWFjaCBnZXR0aW5n IDggaW4gdGhlIGN1cnJlbnQgY29kZS4gVGhlCj4+Pj4+PiBWTUlELXRvLVBBU0lEIG1hcHBpbmcg cmVnaXN0ZXJzIGFyZSB1cGRhdGVkIGJ5IHRoZSBIVyBzY2hlZHVsZXIgd2hlbiB1c2VkLAo+Pj4+ Pj4gYW5kIGJ5IGRyaXZlciBjb2RlIGlmIEhXIHNjaGVkdWxpbmcgaXMgbm90IGJlaW5nIHVzZWQu Cj4+Pj4+PiBUaGUgU2VhIElzbGFuZHMgY29tcHV0ZSBxdWV1ZXMgdXNlIGEgbmV3ICJkb29yYmVs bCIgbWVjaGFuaXNtIGluc3RlYWQgb2YgdGhlCj4+Pj4+PiBlYXJsaWVyIGtlcm5lbC1tYW5hZ2Vk IHdyaXRlIHBvaW50ZXIgcmVnaXN0ZXJzLiBEb29yYmVsbHMgdXNlIGEgc2VwYXJhdGUgQkFSCj4+ Pj4+PiBkZWRpY2F0ZWQgZm9yIHRoaXMgcHVycG9zZSwgYW5kIHBhZ2VzIHdpdGhpbiB0aGUgZG9v cmJlbGwgYXBlcnR1cmUgYXJlCj4+Pj4+PiBtYXBwZWQgdG8gdXNlcnNwYWNlIChlYWNoIHBhZ2Ug bWFwcGVkIHRvIG9ubHkgb25lIHVzZXIgYWRkcmVzcyBzcGFjZSkuCj4+Pj4+PiBXcml0ZXMgdG8g dGhlIGRvb3JiZWxsIGFwZXJ0dXJlIGFyZSBpbnRlcmNlcHRlZCBieSBHUFUgaGFyZHdhcmUsIGFs bG93aW5nCj4+Pj4+PiB1c2Vyc3BhY2UgY29kZSB0byBzYWZlbHkgbWFuYWdlIHdvcmsgcXVldWVz IChyaW5ncykgd2l0aG91dCByZXF1aXJpbmcgYQo+Pj4+Pj4ga2VybmVsIGNhbGwgZm9yIGV2ZXJ5 IHJpbmcgdXBkYXRlLgo+Pj4+Pj4gRmlyc3Qgc3RlcCBmb3IgYW4gYXBwbGljYXRpb24gcHJvY2Vz cyBpcyB0byBvcGVuIHRoZSBrZmQgZGV2aWNlLiBDYWxscyB0bwo+Pj4+Pj4gb3BlbiBjcmVhdGUg YSBrZmQgInByb2Nlc3MiIHN0cnVjdHVyZSBvbmx5IGZvciB0aGUgZmlyc3QgdGhyZWFkIG9mIHRo ZQo+Pj4+Pj4gcHJvY2Vzcy4gU3Vic2VxdWVudCBvcGVuIGNhbGxzIGFyZSBjaGVja2VkIHRvIHNl ZSBpZiB0aGV5IGFyZSBmcm9tIHByb2Nlc3Nlcwo+Pj4+Pj4gdXNpbmcgdGhlIHNhbWUgbW1fc3Ry dWN0IGFuZCwgaWYgc28sIGRvbid0IGRvIGFueXRoaW5nLiBUaGUga2ZkIHBlci1wcm9jZXNzCj4+ Pj4+PiBkYXRhIGxpdmVzIGFzIGxvbmcgYXMgdGhlIG1tX3N0cnVjdCBleGlzdHMuIEVhY2ggbW1f c3RydWN0IGlzIGFzc29jaWF0ZWQKPj4+Pj4+IHdpdGggYSB1bmlxdWUgUEFTSUQsIGFsbG93aW5n IHRoZSBJT01NVXYyIHRvIG1ha2UgdXNlcnNwYWNlIHByb2Nlc3MgbWVtb3J5Cj4+Pj4+PiBhY2Nl c3NpYmxlIHRvIHRoZSBHUFUuCj4+Pj4+PiBOZXh0IHN0ZXAgaXMgZm9yIHRoZSBhcHBsaWNhdGlv biB0byBjb2xsZWN0IHRvcG9sb2d5IGluZm9ybWF0aW9uIHZpYSBzeXNmcy4KPj4+Pj4+IFRoaXMg Z2l2ZXMgdXNlcnNwYWNlIGVub3VnaCBpbmZvcm1hdGlvbiB0byBiZSBhYmxlIHRvIGlkZW50aWZ5 IHNwZWNpZmljCj4+Pj4+PiBub2RlcyAocHJvY2Vzc29ycykgaW4gc3Vic2VxdWVudCBxdWV1ZSBt YW5hZ2VtZW50IGNhbGxzLiBBcHBsaWNhdGlvbgo+Pj4+Pj4gcHJvY2Vzc2VzIGNhbiBjcmVhdGUg cXVldWVzIG9uIG11bHRpcGxlIHByb2Nlc3NvcnMsIGFuZCBwcm9jZXNzb3JzIHN1cHBvcnQKPj4+ Pj4+IHF1ZXVlcyBmcm9tIG11bHRpcGxlIHByb2Nlc3Nlcy4KPj4+Pj4+IEF0IHRoaXMgcG9pbnQg dGhlIGFwcGxpY2F0aW9uIGNhbiBjcmVhdGUgd29yayBxdWV1ZXMgaW4gdXNlcnNwYWNlIG1lbW9y eSBhbmQKPj4+Pj4+IHBhc3MgdGhlbSB0aHJvdWdoIHRoZSB1c2VybW9kZSBsaWJyYXJ5IHRvIGtm ZCB0byBoYXZlIHRoZW0gbWFwcGVkIG9udG8gSFcKPj4+Pj4+IHF1ZXVlIHNsb3RzIHNvIHRoYXQg Y29tbWFuZHMgd3JpdHRlbiB0byB0aGUgcXVldWVzIGNhbiBiZSBleGVjdXRlZCBieSB0aGUKPj4+ Pj4+IEdQVS4gUXVldWUgb3BlcmF0aW9ucyBzcGVjaWZ5IGEgcHJvY2Vzc29yIG5vZGUsIGFuZCBz byB0aGUgYnVsayBvZiB0aGlzIGNvZGUKPj4+Pj4+IGlzIGRldmljZS1zcGVjaWZpYy4KPj4+Pj4+ IFdyaXR0ZW4gYnkgSm9obiBCcmlkZ21hbiA8Sm9obi5CcmlkZ21hbkBhbWQuY29tPgo+Pj4+Pj4K Pj4+Pj4+Cj4+Pj4+PiBBbGV4ZXkgU2tpZGFub3YgKDEpOgo+Pj4+Pj4gICBhbWRrZmQ6IEltcGxl bWVudCB0aGUgR2V0IFByb2Nlc3MgQXBlcnR1cmUgSU9DVEwKPj4+Pj4+Cj4+Pj4+PiBBbmRyZXcg TGV3eWNreSAoMyk6Cj4+Pj4+PiAgIGFtZGtmZDogQWRkIGJhc2ljIG1vZHVsZXMgdG8gYW1ka2Zk Cj4+Pj4+PiAgIGFtZGtmZDogQWRkIGludGVycnVwdCBoYW5kbGluZyBtb2R1bGUKPj4+Pj4+ICAg YW1ka2ZkOiBJbXBsZW1lbnQgdGhlIFNldCBNZW1vcnkgUG9saWN5IElPQ1RMCj4+Pj4+Pgo+Pj4+ Pj4gQmVuIEdveiAoOCk6Cj4+Pj4+PiAgIGFtZGtmZDogQWRkIHF1ZXVlIG1vZHVsZQo+Pj4+Pj4g ICBhbWRrZmQ6IEFkZCBtcWRfbWFuYWdlciBtb2R1bGUKPj4+Pj4+ICAgYW1ka2ZkOiBBZGQga2Vy bmVsIHF1ZXVlIG1vZHVsZQo+Pj4+Pj4gICBhbWRrZmQ6IEFkZCBtb2R1bGUgcGFyYW1ldGVyIG9m IHNjaGVkdWxpbmcgcG9saWN5Cj4+Pj4+PiAgIGFtZGtmZDogQWRkIHBhY2tldCBtYW5hZ2VyIG1v ZHVsZQo+Pj4+Pj4gICBhbWRrZmQ6IEFkZCBwcm9jZXNzIHF1ZXVlIG1hbmFnZXIgbW9kdWxlCj4+ Pj4+PiAgIGFtZGtmZDogQWRkIGRldmljZSBxdWV1ZSBtYW5hZ2VyIG1vZHVsZQo+Pj4+Pj4gICBh bWRrZmQ6IEltcGxlbWVudCB0aGUgY3JlYXRlL2Rlc3Ryb3kvdXBkYXRlIHF1ZXVlIElPQ1RMcwo+ Pj4+Pj4KPj4+Pj4+IEV2Z2VueSBQaW5jaHVrICgzKToKPj4+Pj4+ICAgYW1ka2ZkOiBBZGQgdG9w b2xvZ3kgbW9kdWxlIHRvIGFtZGtmZAo+Pj4+Pj4gICBhbWRrZmQ6IEltcGxlbWVudCB0aGUgR2V0 IENsb2NrIENvdW50ZXJzIElPQ1RMCj4+Pj4+PiAgIGFtZGtmZDogSW1wbGVtZW50IHRoZSBQTUMg QWNxdWlyZS9SZWxlYXNlIElPQ1RMcwo+Pj4+Pj4KPj4+Pj4+IE9kZWQgR2FiYmF5ICgxMCk6Cj4+ Pj4+PiAgIG1tOiBBZGQga2ZkX3Byb2Nlc3MgcG9pbnRlciB0byBtbV9zdHJ1Y3QKPj4+Pj4+ICAg ZHJtL3JhZGVvbjogcmVkdWNlIG51bWJlciBvZiBmcmVlIFZNSURzIGFuZCBwaXBlcyBpbiBLVgo+ Pj4+Pj4gICBkcm0vcmFkZW9uL2NpazogRG9uJ3QgdG91Y2ggaW50IG9mIHBpcGVzIDEtNwo+Pj4+ Pj4gICBkcm0vcmFkZW9uOiBSZXBvcnQgZG9vcmJlbGwgY29uZmlndXJhdGlvbiB0byBhbWRrZmQK Pj4+Pj4+ICAgZHJtL3JhZGVvbjogYWRkaW5nIHN5bmNocm9uaXphdGlvbiBmb3IgR1JCTSBHRlgK Pj4+Pj4+ICAgZHJtL3JhZGVvbjogQWRkIHJhZGVvbiA8LS0+IGFtZGtmZCBpbnRlcmZhY2UKPj4+ Pj4+ICAgVXBkYXRlIE1BSU5UQUlORVJTIGFuZCBDUkVESVRTIGZpbGVzIHdpdGggYW1ka2ZkIGlu Zm8KPj4+Pj4+ICAgYW1ka2ZkOiBBZGQgSU9DVEwgc2V0IGRlZmluaXRpb25zIG9mIGFtZGtmZAo+ Pj4+Pj4gICBhbWRrZmQ6IEFkZCBhbWRrZmQgc2tlbGV0b24gZHJpdmVyCj4+Pj4+PiAgIGFtZGtm ZDogQWRkIGJpbmRpbmcvdW5iaW5kaW5nIGNhbGxzIHRvIGFtZF9pb21tdSBkcml2ZXIKPj4+Pj4+ Cj4+Pj4+PiAgQ1JFRElUUyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgfCAgICA3ICsKPj4+Pj4+ICBNQUlOVEFJTkVSUyAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB8ICAgMTAgKwo+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vS2Nv bmZpZyAgICAgICAgICAgICAgICAgICAgIHwgICAgMiArCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJt L3JhZGVvbi9NYWtlZmlsZSAgICAgICAgICAgICAgICAgICAgfCAgICAzICsKPj4+Pj4+ICBkcml2 ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9LY29uZmlnICAgICAgICAgICAgICB8ICAgMTAgKwo+ Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL01ha2VmaWxlICAgICAgICAgICAg IHwgICAxNCArCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQvY2lrX21xZHMu aCAgICAgICAgICAgfCAgMTg1ICsrKwo+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1k a2ZkL2Npa19yZWdzLmggICAgICAgICAgIHwgIDIyMCArKysrCj4+Pj4+PiAgZHJpdmVycy9ncHUv ZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2FwZXJ0dXJlLmMgICAgICAgfCAgMTIzICsrCj4+Pj4+PiAg ZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2NoYXJkZXYuYyAgICAgICAgfCAgNTE4 ICsrKysrKysrKwo+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF9jcmF0 LmggICAgICAgICAgIHwgIDI5NCArKysrKwo+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24v YW1ka2ZkL2tmZF9kZXZpY2UuYyAgICAgICAgIHwgIDI1NCArKysrCj4+Pj4+PiAgLi4uL2RybS9y YWRlb24vYW1ka2ZkL2tmZF9kZXZpY2VfcXVldWVfbWFuYWdlci5jICAgfCAgOTg1ICsrKysrKysr KysrKysrKysKPj4+Pj4+ICAuLi4vZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2RldmljZV9xdWV1ZV9t YW5hZ2VyLmggICB8ICAxMDEgKysKPj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtm ZC9rZmRfZG9vcmJlbGwuYyAgICAgICB8ICAyNjQgKysrKysKPj4+Pj4+ICBkcml2ZXJzL2dwdS9k cm0vcmFkZW9uL2FtZGtmZC9rZmRfaW50ZXJydXB0LmMgICAgICB8ICAxNjEgKysrCj4+Pj4+PiAg ZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2tlcm5lbF9xdWV1ZS5jICAgfCAgMzA1 ICsrKysrCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2tlcm5lbF9x dWV1ZS5oICAgfCAgIDY2ICsrCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQv a2ZkX21vZHVsZS5jICAgICAgICAgfCAgMTMxICsrKwo+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9y YWRlb24vYW1ka2ZkL2tmZF9tcWRfbWFuYWdlci5jICAgIHwgIDI5MSArKysrKwo+Pj4+Pj4gIGRy aXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF9tcWRfbWFuYWdlci5oICAgIHwgICA1NCAr Cj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3BhY2tldF9tYW5hZ2Vy LmMgfCAgNDg4ICsrKysrKysrCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQv a2ZkX3Bhc2lkLmMgICAgICAgICAgfCAgIDk3ICsrCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3Jh ZGVvbi9hbWRrZmQva2ZkX3BtNF9oZWFkZXJzLmggICAgfCAgNjgyICsrKysrKysrKysrCj4+Pj4+ PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3BtNF9vcGNvZGVzLmggICAgfCAg MTA3ICsrCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3ByaXYuaCAg ICAgICAgICAgfCAgNDY2ICsrKysrKysrCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9h bWRrZmQva2ZkX3Byb2Nlc3MuYyAgICAgICAgfCAgNDA1ICsrKysrKysKPj4+Pj4+ICAuLi4vZHJt L3JhZGVvbi9hbWRrZmQva2ZkX3Byb2Nlc3NfcXVldWVfbWFuYWdlci5jICB8ICAzNDMgKysrKysr Cj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3F1ZXVlLmMgICAgICAg ICAgfCAgMTA5ICsrCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3Rv cG9sb2d5LmMgICAgICAgfCAxMjA3Cj4+Pj4+PiArKysrKysrKysrKysrKysrKysrKwo+Pj4+Pj4g IGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF90b3BvbG9neS5oICAgICAgIHwgIDE2 OCArKysKPj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfdmlkbWVtLmMg ICAgICAgICB8ICAgOTYgKysKPj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2Npay5jICAg ICAgICAgICAgICAgICAgICAgICB8ICAxNTQgKy0tCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3Jh ZGVvbi9jaWtfcmVnLmggICAgICAgICAgICAgICAgICAgfCAgIDY1ICsrCj4+Pj4+PiAgZHJpdmVy cy9ncHUvZHJtL3JhZGVvbi9jaWtkLmggICAgICAgICAgICAgICAgICAgICAgfCAgIDUxICstCj4+ Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9yYWRlb24uaCAgICAgICAgICAgICAgICAgICAg fCAgICA5ICsKPj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL3JhZGVvbl9kZXZpY2UuYyAg ICAgICAgICAgICB8ICAgMzIgKwo+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vcmFkZW9u X2Rydi5jICAgICAgICAgICAgICAgIHwgICAgNSArCj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3Jh ZGVvbi9yYWRlb25fa2ZkLmMgICAgICAgICAgICAgICAgfCAgNTY2ICsrKysrKysrKwo+Pj4+Pj4g IGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vcmFkZW9uX2tmZC5oICAgICAgICAgICAgICAgIHwgIDEx OSArKwo+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vcmFkZW9uX2ttcy5jICAgICAgICAg ICAgICAgIHwgICAgNyArCj4+Pj4+PiAgaW5jbHVkZS9saW51eC9tbV90eXBlcy5oICAgICAgICAg ICAgICAgICAgICAgICAgICAgfCAgIDE0ICsKPj4+Pj4+ICBpbmNsdWRlL3VhcGkvbGludXgva2Zk X2lvY3RsLmggICAgICAgICAgICAgICAgICAgICB8ICAxMzMgKysrCj4+Pj4+PiAgNDMgZmlsZXMg Y2hhbmdlZCwgOTIyNiBpbnNlcnRpb25zKCspLCA5NSBkZWxldGlvbnMoLSkKPj4+Pj4+ICBjcmVh dGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQvS2NvbmZpZwo+Pj4+ Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9NYWtl ZmlsZQo+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2Ft ZGtmZC9jaWtfbXFkcy5oCj4+Pj4+PiAgY3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2Ry bS9yYWRlb24vYW1ka2ZkL2Npa19yZWdzLmgKPj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJp dmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2FwZXJ0dXJlLmMKPj4+Pj4+ICBjcmVhdGUg bW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2NoYXJkZXYuYwo+ Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9r ZmRfY3JhdC5oCj4+Pj4+PiAgY3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9yYWRl b24vYW1ka2ZkL2tmZF9kZXZpY2UuYwo+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJz L2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfZGV2aWNlX3F1ZXVlX21hbmFnZXIuYwo+Pj4+Pj4g IGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfZGV2 aWNlX3F1ZXVlX21hbmFnZXIuaAo+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dw dS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfZG9vcmJlbGwuYwo+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEw MDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfaW50ZXJydXB0LmMKPj4+Pj4+ ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2tl cm5lbF9xdWV1ZS5jCj4+Pj4+PiAgY3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9y YWRlb24vYW1ka2ZkL2tmZF9rZXJuZWxfcXVldWUuaAo+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0 NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfbW9kdWxlLmMKPj4+Pj4+ICBjcmVh dGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX21xZF9tYW5h Z2VyLmMKPj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9h bWRrZmQva2ZkX21xZF9tYW5hZ2VyLmgKPj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVy cy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3BhY2tldF9tYW5hZ2VyLmMKPj4+Pj4+ICBjcmVh dGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3Bhc2lkLmMK Pj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQv a2ZkX3BtNF9oZWFkZXJzLmgKPj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUv ZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3BtNF9vcGNvZGVzLmgKPj4+Pj4+ICBjcmVhdGUgbW9kZSAx MDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3ByaXYuaAo+Pj4+Pj4gIGNy ZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfcHJvY2Vz cy5jCj4+Pj4+PiAgY3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1k a2ZkL2tmZF9wcm9jZXNzX3F1ZXVlX21hbmFnZXIuYwo+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0 NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfcXVldWUuYwo+Pj4+Pj4gIGNyZWF0 ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfdG9wb2xvZ3ku Ywo+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtm ZC9rZmRfdG9wb2xvZ3kuaAo+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9k cm0vcmFkZW9uL2FtZGtmZC9rZmRfdmlkbWVtLmMKPj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQg ZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9yYWRlb25fa2ZkLmMKPj4+Pj4+ICBjcmVhdGUgbW9kZSAx MDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9yYWRlb25fa2ZkLmgKPj4+Pj4+ICBjcmVhdGUg bW9kZSAxMDA2NDQgaW5jbHVkZS91YXBpL2xpbnV4L2tmZF9pb2N0bC5oCj4+Pj4+Pgo+Pj4+Pj4g LS0KPj4+Pj4+IDEuOS4xCj4+Pj4+Pgo+Pj4+Cj4+Pgo+PgoKX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KZHJpLWRldmVsIG1haWxpbmcgbGlzdApkcmktZGV2 ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHA6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFp bG1hbi9saXN0aW5mby9kcmktZGV2ZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f51.google.com (mail-pa0-f51.google.com [209.85.220.51]) by kanga.kvack.org (Postfix) with ESMTP id BF8226B0082 for ; Mon, 21 Jul 2014 13:43:12 -0400 (EDT) Received: by mail-pa0-f51.google.com with SMTP id ey11so10307475pad.10 for ; Mon, 21 Jul 2014 10:43:12 -0700 (PDT) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0206.outbound.protection.outlook.com. [207.46.163.206]) by mx.google.com with ESMTPS id s1si5419839pdi.266.2014.07.21.10.43.11 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 21 Jul 2014 10:43:11 -0700 (PDT) Message-ID: <53CD5122.5040804@amd.com> Date: Mon, 21 Jul 2014 20:42:58 +0300 From: Oded Gabbay MIME-Version: 1.0 Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver References: <53C7D645.3070607@amd.com> <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <53CD1FB6.1000602@amd.com> <20140721155437.GA4519@gmail.com> In-Reply-To: <20140721155437.GA4519@gmail.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Jerome Glisse Cc: =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , David Airlie , Alex Deucher , Andrew Morton , John Bridgman , Joerg Roedel , Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , Ben Goz , Alexey Skidanov , Evgeny Pinchuk , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm On 21/07/14 18:54, Jerome Glisse wrote: > On Mon, Jul 21, 2014 at 05:12:06PM +0300, Oded Gabbay wrote: >> On 21/07/14 16:39, Christian K=C3=B6nig wrote: >>> Am 21.07.2014 14:36, schrieb Oded Gabbay: >>>> On 20/07/14 20:46, Jerome Glisse wrote: >>>>> On Thu, Jul 17, 2014 at 04:57:25PM +0300, Oded Gabbay wrote: >>>>>> Forgot to cc mailing list on cover letter. Sorry. >>>>>> >>>>>> As a continuation to the existing discussion, here is a v2 patch s= eries >>>>>> restructured with a cleaner history and no totally-different-early= -versions >>>>>> of the code. >>>>>> >>>>>> Instead of 83 patches, there are now a total of 25 patches, where = 5 of them >>>>>> are modifications to radeon driver and 18 of them include only amd= kfd code. >>>>>> There is no code going away or even modified between patches, only= added. >>>>>> >>>>>> The driver was renamed from radeon_kfd to amdkfd and moved to resi= de under >>>>>> drm/radeon/amdkfd. This move was done to emphasize the fact that t= his driver >>>>>> is an AMD-only driver at this point. Having said that, we do fores= ee a >>>>>> generic hsa framework being implemented in the future and in that = case, we >>>>>> will adjust amdkfd to work within that framework. >>>>>> >>>>>> As the amdkfd driver should support multiple AMD gfx drivers, we w= ant to >>>>>> keep it as a seperate driver from radeon. Therefore, the amdkfd co= de is >>>>>> contained in its own folder. The amdkfd folder was put under the r= adeon >>>>>> folder because the only AMD gfx driver in the Linux kernel at this= point >>>>>> is the radeon driver. Having said that, we will probably need to m= ove it >>>>>> (maybe to be directly under drm) after we integrate with additiona= l AMD gfx >>>>>> drivers. >>>>>> >>>>>> For people who like to review using git, the v2 patch set is locat= ed at: >>>>>> http://cgit.freedesktop.org/~gabbayo/linux/log/?h=3Dkfd-next-3.17-= v2 >>>>>> >>>>>> Written by Oded Gabbayh >>>>> >>>>> So quick comments before i finish going over all patches. There is = many >>>>> things that need more documentation espacialy as of right now there= is >>>>> no userspace i can go look at. >>>> So quick comments on some of your questions but first of all, thanks= for the >>>> time you dedicated to review the code. >>>>> >>>>> There few show stopper, biggest one is gpu memory pinning this is a= big >>>>> no, that would need serious arguments for any hope of convincing me= on >>>>> that side. >>>> We only do gpu memory pinning for kernel objects. There are no users= pace >>>> objects that are pinned on the gpu memory in our driver. If that is = the case, >>>> is it still a show stopper ? >>>> >>>> The kernel objects are: >>>> - pipelines (4 per device) >>>> - mqd per hiq (only 1 per device) >>>> - mqd per userspace queue. On KV, we support up to 1K queues per pro= cess, for >>>> a total of 512K queues. Each mqd is 151 bytes, but the allocation is= done in >>>> 256 alignment. So total *possible* memory is 128MB >>>> - kernel queue (only 1 per device) >>>> - fence address for kernel queue >>>> - runlists for the CP (1 or 2 per device) >>> >>> The main questions here are if it's avoid able to pin down the memory= and if the >>> memory is pinned down at driver load, by request from userspace or by= anything >>> else. >>> >>> As far as I can see only the "mqd per userspace queue" might be a bit >>> questionable, everything else sounds reasonable. >>> >>> Christian. >> >> Most of the pin downs are done on device initialization. >> The "mqd per userspace" is done per userspace queue creation. However,= as I >> said, it has an upper limit of 128MB on KV, and considering the 2G loc= al >> memory, I think it is OK. >> The runlists are also done on userspace queue creation/deletion, but w= e only >> have 1 or 2 runlists per device, so it is not that bad. >=20 > 2G local memory ? You can not assume anything on userside configuration= some > one might build an hsa computer with 512M and still expect a functionin= g > desktop. First of all, I'm only considering Kaveri computer, not "hsa" computer. Second, I would imagine we can build some protection around it, like checking total local memory and limit number of queues based on some percentage of that total local memory. So, if someone will have only 512M, he will be able to open less queues. >=20 > I need to go look into what all this mqd is for, what it does and what = it is > about. But pinning is really bad and this is an issue with userspace co= mmand > scheduling an issue that obviously AMD fails to take into account in de= sign > phase. Maybe, but that is the H/W design non-the-less. We can't very well change the H/W. Oded >=20 >> Oded >>> >>>>> >>>>> It might be better to add a drivers/gpu/drm/amd directory and add c= ommon >>>>> stuff there. >>>>> >>>>> Given that this is not intended to be final HSA api AFAICT then i w= ould >>>>> say this far better to avoid the whole kfd module and add ioctl to = radeon. >>>>> This would avoid crazy communication btw radeon and kfd. >>>>> >>>>> The whole aperture business needs some serious explanation. Especia= ly as >>>>> you want to use userspace address there is nothing to prevent users= pace >>>>> program from allocating things at address you reserve for lds, scra= tch, >>>>> ... only sane way would be to move those lds, scratch inside the vi= rtual >>>>> address reserved for kernel (see kernel memory map). >>>>> >>>>> The whole business of locking performance counter for exclusive per= process >>>>> access is a big NO. Which leads me to the questionable usefullness = of user >>>>> space command ring. >>>> That's like saying: "Which leads me to the questionable usefulness o= f HSA". I >>>> find it analogous to a situation where a network maintainer nacking = a driver >>>> for a network card, which is slower than a different network card. D= oesn't >>>> seem reasonable this situation is would happen. He would still put b= oth the >>>> drivers in the kernel because people want to use the H/W and its fea= tures. So, >>>> I don't think this is a valid reason to NACK the driver. >=20 > Let me rephrase, drop the the performance counter ioctl and modulo memo= ry pinning > i see no objection. In other word, i am not NACKING whole patchset i am= NACKING > the performance ioctl. >=20 > Again this is another argument for round trip to the kernel. As inside = kernel you > could properly do exclusive gpu counter access accross single user cmd = buffer > execution. >=20 >>>> >>>>> I only see issues with that. First and foremost i would >>>>> need to see solid figures that kernel ioctl or syscall has a higher= an >>>>> overhead that is measurable in any meaning full way against a simpl= e >>>>> function call. I know the userspace command ring is a big marketing= features >>>>> that please ignorant userspace programmer. But really this only bri= ngs issues >>>>> and for absolutely not upside afaict. >>>> Really ? You think that doing a context switch to kernel space, with= all its >>>> overhead, is _not_ more expansive than just calling a function in us= erspace >>>> which only puts a buffer on a ring and writes a doorbell ? >=20 > I am saying the overhead is not that big and it probably will not matte= r in most > usecase. For instance i did wrote the most useless kernel module that a= dd two > number through an ioctl (http://people.freedesktop.org/~glisse/adder.ta= r) and > it takes ~0.35microseconds with ioctl while function is ~0.025microseco= nds so > ioctl is 13 times slower. >=20 > Now if there is enough data that shows that a significant percentage of= jobs > submited to the GPU will take less that 0.35microsecond then yes usersp= ace > scheduling does make sense. But so far all we have is handwaving with n= o data > to support any facts. >=20 >=20 > Now if we want to schedule from userspace than you will need to do some= thing > about the pinning, something that gives control to kernel so that kerne= l can > unpin when it wants and move object when it wants no matter what usersp= ace is > doing. >=20 >>>>> >>>>> So i would rather see a very simple ioctl that write the doorbell a= nd might >>>>> do more than that in case of ring/queue overcommit where it would f= irst have >>>>> to wait for a free ring/queue to schedule stuff. This would also al= low sane >>>>> implementation of things like performance counter that could be acq= uire by >>>>> kernel for duration of a job submitted by userspace. While still no= t optimal >>>>> this would be better that userspace locking. >>>>> >>>>> >>>>> I might have more thoughts once i am done with all the patches. >>>>> >>>>> Cheers, >>>>> J=C3=A9r=C3=B4me >>>>> >>>>>> >>>>>> Original Cover Letter: >>>>>> >>>>>> This patch set implements a Heterogeneous System Architecture (HSA= ) driver >>>>>> for radeon-family GPUs. >>>>>> HSA allows different processor types (CPUs, DSPs, GPUs, etc..) to = share >>>>>> system resources more effectively via HW features including shared= pageable >>>>>> memory, userspace-accessible work queues, and platform-level atomi= cs. In >>>>>> addition to the memory protection mechanisms in GPUVM and IOMMUv2,= the Sea >>>>>> Islands family of GPUs also performs HW-level validation of comman= ds passed >>>>>> in through the queues (aka rings). >>>>>> >>>>>> The code in this patch set is intended to serve both as a sample d= river for >>>>>> other HSA-compatible hardware devices and as a production driver f= or >>>>>> radeon-family processors. The code is architected to support multi= ple CPUs >>>>>> each with connected GPUs, although the current implementation focu= ses on a >>>>>> single Kaveri/Berlin APU, and works alongside the existing radeon = kernel >>>>>> graphics driver (kgd). >>>>>> AMD GPUs designed for use with HSA (Sea Islands and up) share some= hardware >>>>>> functionality between HSA compute and regular gfx/compute (memory, >>>>>> interrupts, registers), while other functionality has been added >>>>>> specifically for HSA compute (hw scheduler for virtualized comput= e rings). >>>>>> All shared hardware is owned by the radeon graphics driver, and an= interface >>>>>> between kfd and kgd allows the kfd to make use of those shared res= ources, >>>>>> while HSA-specific functionality is managed directly by kfd by sub= mitting >>>>>> packets into an HSA-specific command queue (the "HIQ"). >>>>>> >>>>>> During kfd module initialization a char device node (/dev/kfd) is = created >>>>>> (surviving until module exit), with ioctls for queue creation & ma= nagement, >>>>>> and data structures are initialized for managing HSA device topolo= gy. >>>>>> The rest of the initialization is driven by calls from the radeon = kgd at the >>>>>> following points : >>>>>> >>>>>> - radeon_init (kfd_init) >>>>>> - radeon_exit (kfd_fini) >>>>>> - radeon_driver_load_kms (kfd_device_probe, kfd_device_init) >>>>>> - radeon_driver_unload_kms (kfd_device_fini) >>>>>> >>>>>> During the probe and init processing per-device data structures ar= e >>>>>> established which connect to the associated graphics kernel driver= . This >>>>>> information is exposed to userspace via sysfs, along with a versio= n number >>>>>> allowing userspace to determine if a topology change has occurred = while it >>>>>> was reading from sysfs. >>>>>> The interface between kfd and kgd also allows the kfd to request b= uffer >>>>>> management services from kgd, and allows kgd to route interrupt re= quests to >>>>>> kfd code since the interrupt block is shared between regular >>>>>> graphics/compute and HSA compute subsystems in the GPU. >>>>>> >>>>>> The kfd code works with an open source usermode library ("libhsakm= t") which >>>>>> is in the final stages of IP review and should be published in a s= eparate >>>>>> repo over the next few days. >>>>>> The code operates in one of three modes, selectable via the sched_= policy >>>>>> module parameter : >>>>>> >>>>>> - sched_policy=3D0 uses a hardware scheduler running in the MEC bl= ock within >>>>>> CP, and allows oversubscription (more queues than HW slots) >>>>>> - sched_policy=3D1 also uses HW scheduling but does not allow >>>>>> oversubscription, so create_queue requests fail when we run out of= HW slots >>>>>> - sched_policy=3D2 does not use HW scheduling, so the driver manua= lly assigns >>>>>> queues to HW slots by programming registers >>>>>> >>>>>> The "no HW scheduling" option is for debug & new hardware bringup = only, so >>>>>> has less test coverage than the other options. Default in the curr= ent code >>>>>> is "HW scheduling without oversubscription" since that is where we= have the >>>>>> most test coverage but we expect to change the default to "HW sche= duling >>>>>> with oversubscription" after further testing. This effectively rem= oves the >>>>>> HW limit on the number of work queues available to applications. >>>>>> >>>>>> Programs running on the GPU are associated with an address space t= hrough the >>>>>> VMID field, which is translated to a unique PASID at access time v= ia a set >>>>>> of 16 VMID-to-PASID mapping registers. The available VMIDs (curren= tly 16) >>>>>> are partitioned (under control of the radeon kgd) between current >>>>>> gfx/compute and HSA compute, with each getting 8 in the current co= de. The >>>>>> VMID-to-PASID mapping registers are updated by the HW scheduler wh= en used, >>>>>> and by driver code if HW scheduling is not being used. >>>>>> The Sea Islands compute queues use a new "doorbell" mechanism inst= ead of the >>>>>> earlier kernel-managed write pointer registers. Doorbells use a se= parate BAR >>>>>> dedicated for this purpose, and pages within the doorbell aperture= are >>>>>> mapped to userspace (each page mapped to only one user address spa= ce). >>>>>> Writes to the doorbell aperture are intercepted by GPU hardware, a= llowing >>>>>> userspace code to safely manage work queues (rings) without requir= ing a >>>>>> kernel call for every ring update. >>>>>> First step for an application process is to open the kfd device. C= alls to >>>>>> open create a kfd "process" structure only for the first thread of= the >>>>>> process. Subsequent open calls are checked to see if they are from= processes >>>>>> using the same mm_struct and, if so, don't do anything. The kfd pe= r-process >>>>>> data lives as long as the mm_struct exists. Each mm_struct is asso= ciated >>>>>> with a unique PASID, allowing the IOMMUv2 to make userspace proces= s memory >>>>>> accessible to the GPU. >>>>>> Next step is for the application to collect topology information v= ia sysfs. >>>>>> This gives userspace enough information to be able to identify spe= cific >>>>>> nodes (processors) in subsequent queue management calls. Applicati= on >>>>>> processes can create queues on multiple processors, and processors= support >>>>>> queues from multiple processes. >>>>>> At this point the application can create work queues in userspace = memory and >>>>>> pass them through the usermode library to kfd to have them mapped = onto HW >>>>>> queue slots so that commands written to the queues can be executed= by the >>>>>> GPU. Queue operations specify a processor node, and so the bulk of= this code >>>>>> is device-specific. >>>>>> Written by John Bridgman >>>>>> >>>>>> >>>>>> Alexey Skidanov (1): >>>>>> amdkfd: Implement the Get Process Aperture IOCTL >>>>>> >>>>>> Andrew Lewycky (3): >>>>>> amdkfd: Add basic modules to amdkfd >>>>>> amdkfd: Add interrupt handling module >>>>>> amdkfd: Implement the Set Memory Policy IOCTL >>>>>> >>>>>> Ben Goz (8): >>>>>> amdkfd: Add queue module >>>>>> amdkfd: Add mqd_manager module >>>>>> amdkfd: Add kernel queue module >>>>>> amdkfd: Add module parameter of scheduling policy >>>>>> amdkfd: Add packet manager module >>>>>> amdkfd: Add process queue manager module >>>>>> amdkfd: Add device queue manager module >>>>>> amdkfd: Implement the create/destroy/update queue IOCTLs >>>>>> >>>>>> Evgeny Pinchuk (3): >>>>>> amdkfd: Add topology module to amdkfd >>>>>> amdkfd: Implement the Get Clock Counters IOCTL >>>>>> amdkfd: Implement the PMC Acquire/Release IOCTLs >>>>>> >>>>>> Oded Gabbay (10): >>>>>> mm: Add kfd_process pointer to mm_struct >>>>>> drm/radeon: reduce number of free VMIDs and pipes in KV >>>>>> drm/radeon/cik: Don't touch int of pipes 1-7 >>>>>> drm/radeon: Report doorbell configuration to amdkfd >>>>>> drm/radeon: adding synchronization for GRBM GFX >>>>>> drm/radeon: Add radeon <--> amdkfd interface >>>>>> Update MAINTAINERS and CREDITS files with amdkfd info >>>>>> amdkfd: Add IOCTL set definitions of amdkfd >>>>>> amdkfd: Add amdkfd skeleton driver >>>>>> amdkfd: Add binding/unbinding calls to amd_iommu driver >>>>>> >>>>>> CREDITS | 7 + >>>>>> MAINTAINERS | 10 + >>>>>> drivers/gpu/drm/radeon/Kconfig | 2 + >>>>>> drivers/gpu/drm/radeon/Makefile | 3 + >>>>>> drivers/gpu/drm/radeon/amdkfd/Kconfig | 10 + >>>>>> drivers/gpu/drm/radeon/amdkfd/Makefile | 14 + >>>>>> drivers/gpu/drm/radeon/amdkfd/cik_mqds.h | 185 +++ >>>>>> drivers/gpu/drm/radeon/amdkfd/cik_regs.h | 220 ++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_aperture.c | 123 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_chardev.c | 518 +++++++= ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_crat.h | 294 +++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_device.c | 254 ++++ >>>>>> .../drm/radeon/amdkfd/kfd_device_queue_manager.c | 985 +++++++= +++++++++ >>>>>> .../drm/radeon/amdkfd/kfd_device_queue_manager.h | 101 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_doorbell.c | 264 +++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_interrupt.c | 161 +++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.c | 305 +++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.h | 66 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_module.c | 131 +++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.c | 291 +++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.h | 54 + >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_packet_manager.c | 488 +++++++= + >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pasid.c | 97 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pm4_headers.h | 682 +++++++= ++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pm4_opcodes.h | 107 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_priv.h | 466 +++++++= + >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_process.c | 405 +++++++ >>>>>> .../drm/radeon/amdkfd/kfd_process_queue_manager.c | 343 ++++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_queue.c | 109 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_topology.c | 1207 >>>>>> ++++++++++++++++++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_topology.h | 168 +++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_vidmem.c | 96 ++ >>>>>> drivers/gpu/drm/radeon/cik.c | 154 +-- >>>>>> drivers/gpu/drm/radeon/cik_reg.h | 65 ++ >>>>>> drivers/gpu/drm/radeon/cikd.h | 51 +- >>>>>> drivers/gpu/drm/radeon/radeon.h | 9 + >>>>>> drivers/gpu/drm/radeon/radeon_device.c | 32 + >>>>>> drivers/gpu/drm/radeon/radeon_drv.c | 5 + >>>>>> drivers/gpu/drm/radeon/radeon_kfd.c | 566 +++++++= ++ >>>>>> drivers/gpu/drm/radeon/radeon_kfd.h | 119 ++ >>>>>> drivers/gpu/drm/radeon/radeon_kms.c | 7 + >>>>>> include/linux/mm_types.h | 14 + >>>>>> include/uapi/linux/kfd_ioctl.h | 133 +++ >>>>>> 43 files changed, 9226 insertions(+), 95 deletions(-) >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/Kconfig >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/Makefile >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/cik_mqds.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/cik_regs.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_aperture.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_chardev.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_crat.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device_queue= _manager.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device_queue= _manager.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_doorbell.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_interrupt.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue= .c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue= .h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_module.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.= c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.= h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_packet_manag= er.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pasid.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pm4_headers.= h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pm4_opcodes.= h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_priv.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_process.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_process_queu= e_manager.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_queue.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_topology.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_topology.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_vidmem.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/radeon_kfd.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/radeon_kfd.h >>>>>> create mode 100644 include/uapi/linux/kfd_ioctl.h >>>>>> >>>>>> -- >>>>>> 1.9.1 >>>>>> >>>> >>> >> -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933420AbaGURnc (ORCPT ); Mon, 21 Jul 2014 13:43:32 -0400 Received: from mail-bl2lp0203.outbound.protection.outlook.com ([207.46.163.203]:22349 "EHLO na01-bl2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932629AbaGURn3 convert rfc822-to-8bit (ORCPT ); Mon, 21 Jul 2014 13:43:29 -0400 X-WSS-ID: 0N92P7S-07-8Q7-02 X-M-MSG: Message-ID: <53CD5122.5040804@amd.com> Date: Mon, 21 Jul 2014 20:42:58 +0300 From: Oded Gabbay Organization: AMD User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Jerome Glisse CC: =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "David Airlie" , Alex Deucher , "Andrew Morton" , John Bridgman , Joerg Roedel , Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , Ben Goz , Alexey Skidanov , Evgeny Pinchuk , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver References: <53C7D645.3070607@amd.com> <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <53CD1FB6.1000602@amd.com> <20140721155437.GA4519@gmail.com> In-Reply-To: <20140721155437.GA4519@gmail.com> Content-Type: text/plain; charset="UTF-8" X-Originating-IP: [10.224.155.153] Content-Transfer-Encoding: 8BIT X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:165.204.84.221;CTRY:US;IPV:NLI;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(6009001)(428002)(24454002)(189002)(199002)(51704005)(51914003)(479174003)(19580405001)(83322001)(84676001)(19580395003)(33656002)(44976005)(83072002)(97736001)(21056001)(15975445006)(85852003)(99396002)(106466001)(47776003)(80316001)(64126003)(68736004)(15202345003)(95666004)(102836001)(4396001)(110136001)(87936001)(50466002)(20776003)(76482001)(23676002)(64706001)(74662001)(31966008)(85306003)(107046002)(92566001)(92726001)(36756003)(80022001)(81542001)(86362001)(54356999)(65816999)(50986999)(46102001)(105586002)(83506001)(93886003)(101416001)(87266999)(76176999)(74502001)(79102001)(77982001)(65956001)(65806001)(81342001);DIR:OUT;SFP:;SCL:1;SRVR:BLUPR02MB132;H:atltwp01.amd.com;FPR:;MLV:sfv;PTR:InfoDomainNonexistent;MX:1;LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0279B3DD0D Authentication-Results: spf=none (sender IP is 165.204.84.221) smtp.mailfrom=Oded.Gabbay@amd.com; X-OriginatorOrg: amd4.onmicrosoft.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 21/07/14 18:54, Jerome Glisse wrote: > On Mon, Jul 21, 2014 at 05:12:06PM +0300, Oded Gabbay wrote: >> On 21/07/14 16:39, Christian König wrote: >>> Am 21.07.2014 14:36, schrieb Oded Gabbay: >>>> On 20/07/14 20:46, Jerome Glisse wrote: >>>>> On Thu, Jul 17, 2014 at 04:57:25PM +0300, Oded Gabbay wrote: >>>>>> Forgot to cc mailing list on cover letter. Sorry. >>>>>> >>>>>> As a continuation to the existing discussion, here is a v2 patch series >>>>>> restructured with a cleaner history and no totally-different-early-versions >>>>>> of the code. >>>>>> >>>>>> Instead of 83 patches, there are now a total of 25 patches, where 5 of them >>>>>> are modifications to radeon driver and 18 of them include only amdkfd code. >>>>>> There is no code going away or even modified between patches, only added. >>>>>> >>>>>> The driver was renamed from radeon_kfd to amdkfd and moved to reside under >>>>>> drm/radeon/amdkfd. This move was done to emphasize the fact that this driver >>>>>> is an AMD-only driver at this point. Having said that, we do foresee a >>>>>> generic hsa framework being implemented in the future and in that case, we >>>>>> will adjust amdkfd to work within that framework. >>>>>> >>>>>> As the amdkfd driver should support multiple AMD gfx drivers, we want to >>>>>> keep it as a seperate driver from radeon. Therefore, the amdkfd code is >>>>>> contained in its own folder. The amdkfd folder was put under the radeon >>>>>> folder because the only AMD gfx driver in the Linux kernel at this point >>>>>> is the radeon driver. Having said that, we will probably need to move it >>>>>> (maybe to be directly under drm) after we integrate with additional AMD gfx >>>>>> drivers. >>>>>> >>>>>> For people who like to review using git, the v2 patch set is located at: >>>>>> http://cgit.freedesktop.org/~gabbayo/linux/log/?h=kfd-next-3.17-v2 >>>>>> >>>>>> Written by Oded Gabbayh >>>>> >>>>> So quick comments before i finish going over all patches. There is many >>>>> things that need more documentation espacialy as of right now there is >>>>> no userspace i can go look at. >>>> So quick comments on some of your questions but first of all, thanks for the >>>> time you dedicated to review the code. >>>>> >>>>> There few show stopper, biggest one is gpu memory pinning this is a big >>>>> no, that would need serious arguments for any hope of convincing me on >>>>> that side. >>>> We only do gpu memory pinning for kernel objects. There are no userspace >>>> objects that are pinned on the gpu memory in our driver. If that is the case, >>>> is it still a show stopper ? >>>> >>>> The kernel objects are: >>>> - pipelines (4 per device) >>>> - mqd per hiq (only 1 per device) >>>> - mqd per userspace queue. On KV, we support up to 1K queues per process, for >>>> a total of 512K queues. Each mqd is 151 bytes, but the allocation is done in >>>> 256 alignment. So total *possible* memory is 128MB >>>> - kernel queue (only 1 per device) >>>> - fence address for kernel queue >>>> - runlists for the CP (1 or 2 per device) >>> >>> The main questions here are if it's avoid able to pin down the memory and if the >>> memory is pinned down at driver load, by request from userspace or by anything >>> else. >>> >>> As far as I can see only the "mqd per userspace queue" might be a bit >>> questionable, everything else sounds reasonable. >>> >>> Christian. >> >> Most of the pin downs are done on device initialization. >> The "mqd per userspace" is done per userspace queue creation. However, as I >> said, it has an upper limit of 128MB on KV, and considering the 2G local >> memory, I think it is OK. >> The runlists are also done on userspace queue creation/deletion, but we only >> have 1 or 2 runlists per device, so it is not that bad. > > 2G local memory ? You can not assume anything on userside configuration some > one might build an hsa computer with 512M and still expect a functioning > desktop. First of all, I'm only considering Kaveri computer, not "hsa" computer. Second, I would imagine we can build some protection around it, like checking total local memory and limit number of queues based on some percentage of that total local memory. So, if someone will have only 512M, he will be able to open less queues. > > I need to go look into what all this mqd is for, what it does and what it is > about. But pinning is really bad and this is an issue with userspace command > scheduling an issue that obviously AMD fails to take into account in design > phase. Maybe, but that is the H/W design non-the-less. We can't very well change the H/W. Oded > >> Oded >>> >>>>> >>>>> It might be better to add a drivers/gpu/drm/amd directory and add common >>>>> stuff there. >>>>> >>>>> Given that this is not intended to be final HSA api AFAICT then i would >>>>> say this far better to avoid the whole kfd module and add ioctl to radeon. >>>>> This would avoid crazy communication btw radeon and kfd. >>>>> >>>>> The whole aperture business needs some serious explanation. Especialy as >>>>> you want to use userspace address there is nothing to prevent userspace >>>>> program from allocating things at address you reserve for lds, scratch, >>>>> ... only sane way would be to move those lds, scratch inside the virtual >>>>> address reserved for kernel (see kernel memory map). >>>>> >>>>> The whole business of locking performance counter for exclusive per process >>>>> access is a big NO. Which leads me to the questionable usefullness of user >>>>> space command ring. >>>> That's like saying: "Which leads me to the questionable usefulness of HSA". I >>>> find it analogous to a situation where a network maintainer nacking a driver >>>> for a network card, which is slower than a different network card. Doesn't >>>> seem reasonable this situation is would happen. He would still put both the >>>> drivers in the kernel because people want to use the H/W and its features. So, >>>> I don't think this is a valid reason to NACK the driver. > > Let me rephrase, drop the the performance counter ioctl and modulo memory pinning > i see no objection. In other word, i am not NACKING whole patchset i am NACKING > the performance ioctl. > > Again this is another argument for round trip to the kernel. As inside kernel you > could properly do exclusive gpu counter access accross single user cmd buffer > execution. > >>>> >>>>> I only see issues with that. First and foremost i would >>>>> need to see solid figures that kernel ioctl or syscall has a higher an >>>>> overhead that is measurable in any meaning full way against a simple >>>>> function call. I know the userspace command ring is a big marketing features >>>>> that please ignorant userspace programmer. But really this only brings issues >>>>> and for absolutely not upside afaict. >>>> Really ? You think that doing a context switch to kernel space, with all its >>>> overhead, is _not_ more expansive than just calling a function in userspace >>>> which only puts a buffer on a ring and writes a doorbell ? > > I am saying the overhead is not that big and it probably will not matter in most > usecase. For instance i did wrote the most useless kernel module that add two > number through an ioctl (http://people.freedesktop.org/~glisse/adder.tar) and > it takes ~0.35microseconds with ioctl while function is ~0.025microseconds so > ioctl is 13 times slower. > > Now if there is enough data that shows that a significant percentage of jobs > submited to the GPU will take less that 0.35microsecond then yes userspace > scheduling does make sense. But so far all we have is handwaving with no data > to support any facts. > > > Now if we want to schedule from userspace than you will need to do something > about the pinning, something that gives control to kernel so that kernel can > unpin when it wants and move object when it wants no matter what userspace is > doing. > >>>>> >>>>> So i would rather see a very simple ioctl that write the doorbell and might >>>>> do more than that in case of ring/queue overcommit where it would first have >>>>> to wait for a free ring/queue to schedule stuff. This would also allow sane >>>>> implementation of things like performance counter that could be acquire by >>>>> kernel for duration of a job submitted by userspace. While still not optimal >>>>> this would be better that userspace locking. >>>>> >>>>> >>>>> I might have more thoughts once i am done with all the patches. >>>>> >>>>> Cheers, >>>>> Jérôme >>>>> >>>>>> >>>>>> Original Cover Letter: >>>>>> >>>>>> This patch set implements a Heterogeneous System Architecture (HSA) driver >>>>>> for radeon-family GPUs. >>>>>> HSA allows different processor types (CPUs, DSPs, GPUs, etc..) to share >>>>>> system resources more effectively via HW features including shared pageable >>>>>> memory, userspace-accessible work queues, and platform-level atomics. In >>>>>> addition to the memory protection mechanisms in GPUVM and IOMMUv2, the Sea >>>>>> Islands family of GPUs also performs HW-level validation of commands passed >>>>>> in through the queues (aka rings). >>>>>> >>>>>> The code in this patch set is intended to serve both as a sample driver for >>>>>> other HSA-compatible hardware devices and as a production driver for >>>>>> radeon-family processors. The code is architected to support multiple CPUs >>>>>> each with connected GPUs, although the current implementation focuses on a >>>>>> single Kaveri/Berlin APU, and works alongside the existing radeon kernel >>>>>> graphics driver (kgd). >>>>>> AMD GPUs designed for use with HSA (Sea Islands and up) share some hardware >>>>>> functionality between HSA compute and regular gfx/compute (memory, >>>>>> interrupts, registers), while other functionality has been added >>>>>> specifically for HSA compute (hw scheduler for virtualized compute rings). >>>>>> All shared hardware is owned by the radeon graphics driver, and an interface >>>>>> between kfd and kgd allows the kfd to make use of those shared resources, >>>>>> while HSA-specific functionality is managed directly by kfd by submitting >>>>>> packets into an HSA-specific command queue (the "HIQ"). >>>>>> >>>>>> During kfd module initialization a char device node (/dev/kfd) is created >>>>>> (surviving until module exit), with ioctls for queue creation & management, >>>>>> and data structures are initialized for managing HSA device topology. >>>>>> The rest of the initialization is driven by calls from the radeon kgd at the >>>>>> following points : >>>>>> >>>>>> - radeon_init (kfd_init) >>>>>> - radeon_exit (kfd_fini) >>>>>> - radeon_driver_load_kms (kfd_device_probe, kfd_device_init) >>>>>> - radeon_driver_unload_kms (kfd_device_fini) >>>>>> >>>>>> During the probe and init processing per-device data structures are >>>>>> established which connect to the associated graphics kernel driver. This >>>>>> information is exposed to userspace via sysfs, along with a version number >>>>>> allowing userspace to determine if a topology change has occurred while it >>>>>> was reading from sysfs. >>>>>> The interface between kfd and kgd also allows the kfd to request buffer >>>>>> management services from kgd, and allows kgd to route interrupt requests to >>>>>> kfd code since the interrupt block is shared between regular >>>>>> graphics/compute and HSA compute subsystems in the GPU. >>>>>> >>>>>> The kfd code works with an open source usermode library ("libhsakmt") which >>>>>> is in the final stages of IP review and should be published in a separate >>>>>> repo over the next few days. >>>>>> The code operates in one of three modes, selectable via the sched_policy >>>>>> module parameter : >>>>>> >>>>>> - sched_policy=0 uses a hardware scheduler running in the MEC block within >>>>>> CP, and allows oversubscription (more queues than HW slots) >>>>>> - sched_policy=1 also uses HW scheduling but does not allow >>>>>> oversubscription, so create_queue requests fail when we run out of HW slots >>>>>> - sched_policy=2 does not use HW scheduling, so the driver manually assigns >>>>>> queues to HW slots by programming registers >>>>>> >>>>>> The "no HW scheduling" option is for debug & new hardware bringup only, so >>>>>> has less test coverage than the other options. Default in the current code >>>>>> is "HW scheduling without oversubscription" since that is where we have the >>>>>> most test coverage but we expect to change the default to "HW scheduling >>>>>> with oversubscription" after further testing. This effectively removes the >>>>>> HW limit on the number of work queues available to applications. >>>>>> >>>>>> Programs running on the GPU are associated with an address space through the >>>>>> VMID field, which is translated to a unique PASID at access time via a set >>>>>> of 16 VMID-to-PASID mapping registers. The available VMIDs (currently 16) >>>>>> are partitioned (under control of the radeon kgd) between current >>>>>> gfx/compute and HSA compute, with each getting 8 in the current code. The >>>>>> VMID-to-PASID mapping registers are updated by the HW scheduler when used, >>>>>> and by driver code if HW scheduling is not being used. >>>>>> The Sea Islands compute queues use a new "doorbell" mechanism instead of the >>>>>> earlier kernel-managed write pointer registers. Doorbells use a separate BAR >>>>>> dedicated for this purpose, and pages within the doorbell aperture are >>>>>> mapped to userspace (each page mapped to only one user address space). >>>>>> Writes to the doorbell aperture are intercepted by GPU hardware, allowing >>>>>> userspace code to safely manage work queues (rings) without requiring a >>>>>> kernel call for every ring update. >>>>>> First step for an application process is to open the kfd device. Calls to >>>>>> open create a kfd "process" structure only for the first thread of the >>>>>> process. Subsequent open calls are checked to see if they are from processes >>>>>> using the same mm_struct and, if so, don't do anything. The kfd per-process >>>>>> data lives as long as the mm_struct exists. Each mm_struct is associated >>>>>> with a unique PASID, allowing the IOMMUv2 to make userspace process memory >>>>>> accessible to the GPU. >>>>>> Next step is for the application to collect topology information via sysfs. >>>>>> This gives userspace enough information to be able to identify specific >>>>>> nodes (processors) in subsequent queue management calls. Application >>>>>> processes can create queues on multiple processors, and processors support >>>>>> queues from multiple processes. >>>>>> At this point the application can create work queues in userspace memory and >>>>>> pass them through the usermode library to kfd to have them mapped onto HW >>>>>> queue slots so that commands written to the queues can be executed by the >>>>>> GPU. Queue operations specify a processor node, and so the bulk of this code >>>>>> is device-specific. >>>>>> Written by John Bridgman >>>>>> >>>>>> >>>>>> Alexey Skidanov (1): >>>>>> amdkfd: Implement the Get Process Aperture IOCTL >>>>>> >>>>>> Andrew Lewycky (3): >>>>>> amdkfd: Add basic modules to amdkfd >>>>>> amdkfd: Add interrupt handling module >>>>>> amdkfd: Implement the Set Memory Policy IOCTL >>>>>> >>>>>> Ben Goz (8): >>>>>> amdkfd: Add queue module >>>>>> amdkfd: Add mqd_manager module >>>>>> amdkfd: Add kernel queue module >>>>>> amdkfd: Add module parameter of scheduling policy >>>>>> amdkfd: Add packet manager module >>>>>> amdkfd: Add process queue manager module >>>>>> amdkfd: Add device queue manager module >>>>>> amdkfd: Implement the create/destroy/update queue IOCTLs >>>>>> >>>>>> Evgeny Pinchuk (3): >>>>>> amdkfd: Add topology module to amdkfd >>>>>> amdkfd: Implement the Get Clock Counters IOCTL >>>>>> amdkfd: Implement the PMC Acquire/Release IOCTLs >>>>>> >>>>>> Oded Gabbay (10): >>>>>> mm: Add kfd_process pointer to mm_struct >>>>>> drm/radeon: reduce number of free VMIDs and pipes in KV >>>>>> drm/radeon/cik: Don't touch int of pipes 1-7 >>>>>> drm/radeon: Report doorbell configuration to amdkfd >>>>>> drm/radeon: adding synchronization for GRBM GFX >>>>>> drm/radeon: Add radeon <--> amdkfd interface >>>>>> Update MAINTAINERS and CREDITS files with amdkfd info >>>>>> amdkfd: Add IOCTL set definitions of amdkfd >>>>>> amdkfd: Add amdkfd skeleton driver >>>>>> amdkfd: Add binding/unbinding calls to amd_iommu driver >>>>>> >>>>>> CREDITS | 7 + >>>>>> MAINTAINERS | 10 + >>>>>> drivers/gpu/drm/radeon/Kconfig | 2 + >>>>>> drivers/gpu/drm/radeon/Makefile | 3 + >>>>>> drivers/gpu/drm/radeon/amdkfd/Kconfig | 10 + >>>>>> drivers/gpu/drm/radeon/amdkfd/Makefile | 14 + >>>>>> drivers/gpu/drm/radeon/amdkfd/cik_mqds.h | 185 +++ >>>>>> drivers/gpu/drm/radeon/amdkfd/cik_regs.h | 220 ++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_aperture.c | 123 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_chardev.c | 518 +++++++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_crat.h | 294 +++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_device.c | 254 ++++ >>>>>> .../drm/radeon/amdkfd/kfd_device_queue_manager.c | 985 ++++++++++++++++ >>>>>> .../drm/radeon/amdkfd/kfd_device_queue_manager.h | 101 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_doorbell.c | 264 +++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_interrupt.c | 161 +++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.c | 305 +++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.h | 66 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_module.c | 131 +++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.c | 291 +++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.h | 54 + >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_packet_manager.c | 488 ++++++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pasid.c | 97 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pm4_headers.h | 682 +++++++++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pm4_opcodes.h | 107 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_priv.h | 466 ++++++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_process.c | 405 +++++++ >>>>>> .../drm/radeon/amdkfd/kfd_process_queue_manager.c | 343 ++++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_queue.c | 109 ++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_topology.c | 1207 >>>>>> ++++++++++++++++++++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_topology.h | 168 +++ >>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_vidmem.c | 96 ++ >>>>>> drivers/gpu/drm/radeon/cik.c | 154 +-- >>>>>> drivers/gpu/drm/radeon/cik_reg.h | 65 ++ >>>>>> drivers/gpu/drm/radeon/cikd.h | 51 +- >>>>>> drivers/gpu/drm/radeon/radeon.h | 9 + >>>>>> drivers/gpu/drm/radeon/radeon_device.c | 32 + >>>>>> drivers/gpu/drm/radeon/radeon_drv.c | 5 + >>>>>> drivers/gpu/drm/radeon/radeon_kfd.c | 566 +++++++++ >>>>>> drivers/gpu/drm/radeon/radeon_kfd.h | 119 ++ >>>>>> drivers/gpu/drm/radeon/radeon_kms.c | 7 + >>>>>> include/linux/mm_types.h | 14 + >>>>>> include/uapi/linux/kfd_ioctl.h | 133 +++ >>>>>> 43 files changed, 9226 insertions(+), 95 deletions(-) >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/Kconfig >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/Makefile >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/cik_mqds.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/cik_regs.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_aperture.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_chardev.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_crat.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device_queue_manager.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device_queue_manager.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_doorbell.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_interrupt.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_module.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_packet_manager.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pasid.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pm4_headers.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pm4_opcodes.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_priv.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_process.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_process_queue_manager.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_queue.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_topology.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_topology.h >>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_vidmem.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/radeon_kfd.c >>>>>> create mode 100644 drivers/gpu/drm/radeon/radeon_kfd.h >>>>>> create mode 100644 include/uapi/linux/kfd_ioctl.h >>>>>> >>>>>> -- >>>>>> 1.9.1 >>>>>> >>>> >>> >>