From mboxrd@z Thu Jan 1 00:00:00 1970 From: Oded Gabbay Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver Date: Tue, 22 Jul 2014 00:56:13 +0300 Message-ID: <53CD8C7D.9010106@amd.com> References: <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <53CD1FB6.1000602@amd.com> <20140721155437.GA4519@gmail.com> <53CD5122.5040804@amd.com> <20140721181433.GA5196@gmail.com> <53CD5DBC.7010301@amd.com> <20140721185940.GA5278@gmail.com> <53CD68BF.4020308@amd.com> <20140721192837.GC5278@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0184.outbound.protection.outlook.com [207.46.163.184]) by gabe.freedesktop.org (Postfix) with ESMTP id A2BDB6E1D4 for ; Mon, 21 Jul 2014 14:56:28 -0700 (PDT) In-Reply-To: <20140721192837.GC5278@gmail.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Jerome Glisse Cc: Andrew Lewycky , linux-mm , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , Evgeny Pinchuk , Alexey Skidanov , Andrew Morton List-Id: dri-devel@lists.freedesktop.org T24gMjEvMDcvMTQgMjI6MjgsIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4gT24gTW9uLCBKdWwgMjEs IDIwMTQgYXQgMTA6MjM6NDNQTSArMDMwMCwgT2RlZCBHYWJiYXkgd3JvdGU6Cj4+IE9uIDIxLzA3 LzE0IDIxOjU5LCBKZXJvbWUgR2xpc3NlIHdyb3RlOgo+Pj4gT24gTW9uLCBKdWwgMjEsIDIwMTQg YXQgMDk6MzY6NDRQTSArMDMwMCwgT2RlZCBHYWJiYXkgd3JvdGU6Cj4+Pj4gT24gMjEvMDcvMTQg MjE6MTQsIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4+Pj4+IE9uIE1vbiwgSnVsIDIxLCAyMDE0IGF0 IDA4OjQyOjU4UE0gKzAzMDAsIE9kZWQgR2FiYmF5IHdyb3RlOgo+Pj4+Pj4gT24gMjEvMDcvMTQg MTg6NTQsIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4+Pj4+Pj4gT24gTW9uLCBKdWwgMjEsIDIwMTQg YXQgMDU6MTI6MDZQTSArMDMwMCwgT2RlZCBHYWJiYXkgd3JvdGU6Cj4+Pj4+Pj4+IE9uIDIxLzA3 LzE0IDE2OjM5LCBDaHJpc3RpYW4gS8O2bmlnIHdyb3RlOgo+Pj4+Pj4+Pj4gQW0gMjEuMDcuMjAx NCAxNDozNiwgc2NocmllYiBPZGVkIEdhYmJheToKPj4+Pj4+Pj4+PiBPbiAyMC8wNy8xNCAyMDo0 NiwgSmVyb21lIEdsaXNzZSB3cm90ZToKPj4+Pj4+Pj4+Pj4gT24gVGh1LCBKdWwgMTcsIDIwMTQg YXQgMDQ6NTc6MjVQTSArMDMwMCwgT2RlZCBHYWJiYXkgd3JvdGU6Cj4+Pj4+Pj4+Pj4+PiBGb3Jn b3QgdG8gY2MgbWFpbGluZyBsaXN0IG9uIGNvdmVyIGxldHRlci4gU29ycnkuCj4+Pj4+Pj4+Pj4+ Pgo+Pj4+Pj4+Pj4+Pj4gQXMgYSBjb250aW51YXRpb24gdG8gdGhlIGV4aXN0aW5nIGRpc2N1c3Np b24sIGhlcmUgaXMgYSB2MiBwYXRjaCBzZXJpZXMKPj4+Pj4+Pj4+Pj4+IHJlc3RydWN0dXJlZCB3 aXRoIGEgY2xlYW5lciBoaXN0b3J5IGFuZCBubyB0b3RhbGx5LWRpZmZlcmVudC1lYXJseS12ZXJz aW9ucwo+Pj4+Pj4+Pj4+Pj4gb2YgdGhlIGNvZGUuCj4+Pj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4+Pj4g SW5zdGVhZCBvZiA4MyBwYXRjaGVzLCB0aGVyZSBhcmUgbm93IGEgdG90YWwgb2YgMjUgcGF0Y2hl cywgd2hlcmUgNSBvZiB0aGVtCj4+Pj4+Pj4+Pj4+PiBhcmUgbW9kaWZpY2F0aW9ucyB0byByYWRl b24gZHJpdmVyIGFuZCAxOCBvZiB0aGVtIGluY2x1ZGUgb25seSBhbWRrZmQgY29kZS4KPj4+Pj4+ Pj4+Pj4+IFRoZXJlIGlzIG5vIGNvZGUgZ29pbmcgYXdheSBvciBldmVuIG1vZGlmaWVkIGJldHdl ZW4gcGF0Y2hlcywgb25seSBhZGRlZC4KPj4+Pj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pj4+PiBUaGUgZHJp dmVyIHdhcyByZW5hbWVkIGZyb20gcmFkZW9uX2tmZCB0byBhbWRrZmQgYW5kIG1vdmVkIHRvIHJl c2lkZSB1bmRlcgo+Pj4+Pj4+Pj4+Pj4gZHJtL3JhZGVvbi9hbWRrZmQuIFRoaXMgbW92ZSB3YXMg ZG9uZSB0byBlbXBoYXNpemUgdGhlIGZhY3QgdGhhdCB0aGlzIGRyaXZlcgo+Pj4+Pj4+Pj4+Pj4g aXMgYW4gQU1ELW9ubHkgZHJpdmVyIGF0IHRoaXMgcG9pbnQuIEhhdmluZyBzYWlkIHRoYXQsIHdl IGRvIGZvcmVzZWUgYQo+Pj4+Pj4+Pj4+Pj4gZ2VuZXJpYyBoc2EgZnJhbWV3b3JrIGJlaW5nIGlt cGxlbWVudGVkIGluIHRoZSBmdXR1cmUgYW5kIGluIHRoYXQgY2FzZSwgd2UKPj4+Pj4+Pj4+Pj4+ IHdpbGwgYWRqdXN0IGFtZGtmZCB0byB3b3JrIHdpdGhpbiB0aGF0IGZyYW1ld29yay4KPj4+Pj4+ Pj4+Pj4+Cj4+Pj4+Pj4+Pj4+PiBBcyB0aGUgYW1ka2ZkIGRyaXZlciBzaG91bGQgc3VwcG9ydCBt dWx0aXBsZSBBTUQgZ2Z4IGRyaXZlcnMsIHdlIHdhbnQgdG8KPj4+Pj4+Pj4+Pj4+IGtlZXAgaXQg YXMgYSBzZXBlcmF0ZSBkcml2ZXIgZnJvbSByYWRlb24uIFRoZXJlZm9yZSwgdGhlIGFtZGtmZCBj b2RlIGlzCj4+Pj4+Pj4+Pj4+PiBjb250YWluZWQgaW4gaXRzIG93biBmb2xkZXIuIFRoZSBhbWRr ZmQgZm9sZGVyIHdhcyBwdXQgdW5kZXIgdGhlIHJhZGVvbgo+Pj4+Pj4+Pj4+Pj4gZm9sZGVyIGJl Y2F1c2UgdGhlIG9ubHkgQU1EIGdmeCBkcml2ZXIgaW4gdGhlIExpbnV4IGtlcm5lbCBhdCB0aGlz IHBvaW50Cj4+Pj4+Pj4+Pj4+PiBpcyB0aGUgcmFkZW9uIGRyaXZlci4gSGF2aW5nIHNhaWQgdGhh dCwgd2Ugd2lsbCBwcm9iYWJseSBuZWVkIHRvIG1vdmUgaXQKPj4+Pj4+Pj4+Pj4+IChtYXliZSB0 byBiZSBkaXJlY3RseSB1bmRlciBkcm0pIGFmdGVyIHdlIGludGVncmF0ZSB3aXRoIGFkZGl0aW9u YWwgQU1EIGdmeAo+Pj4+Pj4+Pj4+Pj4gZHJpdmVycy4KPj4+Pj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pj4+ PiBGb3IgcGVvcGxlIHdobyBsaWtlIHRvIHJldmlldyB1c2luZyBnaXQsIHRoZSB2MiBwYXRjaCBz ZXQgaXMgbG9jYXRlZCBhdDoKPj4+Pj4+Pj4+Pj4+IGh0dHA6Ly9jZ2l0LmZyZWVkZXNrdG9wLm9y Zy9+Z2FiYmF5by9saW51eC9sb2cvP2g9a2ZkLW5leHQtMy4xNy12Mgo+Pj4+Pj4+Pj4+Pj4KPj4+ Pj4+Pj4+Pj4+IFdyaXR0ZW4gYnkgT2RlZCBHYWJiYXloIDxvZGVkLmdhYmJheUBhbWQuY29tPgo+ Pj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4+PiBTbyBxdWljayBjb21tZW50cyBiZWZvcmUgaSBmaW5pc2gg Z29pbmcgb3ZlciBhbGwgcGF0Y2hlcy4gVGhlcmUgaXMgbWFueQo+Pj4+Pj4+Pj4+PiB0aGluZ3Mg dGhhdCBuZWVkIG1vcmUgZG9jdW1lbnRhdGlvbiBlc3BhY2lhbHkgYXMgb2YgcmlnaHQgbm93IHRo ZXJlIGlzCj4+Pj4+Pj4+Pj4+IG5vIHVzZXJzcGFjZSBpIGNhbiBnbyBsb29rIGF0Lgo+Pj4+Pj4+ Pj4+IFNvIHF1aWNrIGNvbW1lbnRzIG9uIHNvbWUgb2YgeW91ciBxdWVzdGlvbnMgYnV0IGZpcnN0 IG9mIGFsbCwgdGhhbmtzIGZvciB0aGUKPj4+Pj4+Pj4+PiB0aW1lIHlvdSBkZWRpY2F0ZWQgdG8g cmV2aWV3IHRoZSBjb2RlLgo+Pj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4+PiBUaGVyZSBmZXcgc2hvdyBz dG9wcGVyLCBiaWdnZXN0IG9uZSBpcyBncHUgbWVtb3J5IHBpbm5pbmcgdGhpcyBpcyBhIGJpZwo+ Pj4+Pj4+Pj4+PiBubywgdGhhdCB3b3VsZCBuZWVkIHNlcmlvdXMgYXJndW1lbnRzIGZvciBhbnkg aG9wZSBvZiBjb252aW5jaW5nIG1lIG9uCj4+Pj4+Pj4+Pj4+IHRoYXQgc2lkZS4KPj4+Pj4+Pj4+ PiBXZSBvbmx5IGRvIGdwdSBtZW1vcnkgcGlubmluZyBmb3Iga2VybmVsIG9iamVjdHMuIFRoZXJl IGFyZSBubyB1c2Vyc3BhY2UKPj4+Pj4+Pj4+PiBvYmplY3RzIHRoYXQgYXJlIHBpbm5lZCBvbiB0 aGUgZ3B1IG1lbW9yeSBpbiBvdXIgZHJpdmVyLiBJZiB0aGF0IGlzIHRoZSBjYXNlLAo+Pj4+Pj4+ Pj4+IGlzIGl0IHN0aWxsIGEgc2hvdyBzdG9wcGVyID8KPj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4+IFRo ZSBrZXJuZWwgb2JqZWN0cyBhcmU6Cj4+Pj4+Pj4+Pj4gLSBwaXBlbGluZXMgKDQgcGVyIGRldmlj ZSkKPj4+Pj4+Pj4+PiAtIG1xZCBwZXIgaGlxIChvbmx5IDEgcGVyIGRldmljZSkKPj4+Pj4+Pj4+ PiAtIG1xZCBwZXIgdXNlcnNwYWNlIHF1ZXVlLiBPbiBLViwgd2Ugc3VwcG9ydCB1cCB0byAxSyBx dWV1ZXMgcGVyIHByb2Nlc3MsIGZvcgo+Pj4+Pj4+Pj4+IGEgdG90YWwgb2YgNTEySyBxdWV1ZXMu IEVhY2ggbXFkIGlzIDE1MSBieXRlcywgYnV0IHRoZSBhbGxvY2F0aW9uIGlzIGRvbmUgaW4KPj4+ Pj4+Pj4+PiAyNTYgYWxpZ25tZW50LiBTbyB0b3RhbCAqcG9zc2libGUqIG1lbW9yeSBpcyAxMjhN Qgo+Pj4+Pj4+Pj4+IC0ga2VybmVsIHF1ZXVlIChvbmx5IDEgcGVyIGRldmljZSkKPj4+Pj4+Pj4+ PiAtIGZlbmNlIGFkZHJlc3MgZm9yIGtlcm5lbCBxdWV1ZQo+Pj4+Pj4+Pj4+IC0gcnVubGlzdHMg Zm9yIHRoZSBDUCAoMSBvciAyIHBlciBkZXZpY2UpCj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4gVGhlIG1h aW4gcXVlc3Rpb25zIGhlcmUgYXJlIGlmIGl0J3MgYXZvaWQgYWJsZSB0byBwaW4gZG93biB0aGUg bWVtb3J5IGFuZCBpZiB0aGUKPj4+Pj4+Pj4+IG1lbW9yeSBpcyBwaW5uZWQgZG93biBhdCBkcml2 ZXIgbG9hZCwgYnkgcmVxdWVzdCBmcm9tIHVzZXJzcGFjZSBvciBieSBhbnl0aGluZwo+Pj4+Pj4+ Pj4gZWxzZS4KPj4+Pj4+Pj4+Cj4+Pj4+Pj4+PiBBcyBmYXIgYXMgSSBjYW4gc2VlIG9ubHkgdGhl ICJtcWQgcGVyIHVzZXJzcGFjZSBxdWV1ZSIgbWlnaHQgYmUgYSBiaXQKPj4+Pj4+Pj4+IHF1ZXN0 aW9uYWJsZSwgZXZlcnl0aGluZyBlbHNlIHNvdW5kcyByZWFzb25hYmxlLgo+Pj4+Pj4+Pj4KPj4+ Pj4+Pj4+IENocmlzdGlhbi4KPj4+Pj4+Pj4KPj4+Pj4+Pj4gTW9zdCBvZiB0aGUgcGluIGRvd25z IGFyZSBkb25lIG9uIGRldmljZSBpbml0aWFsaXphdGlvbi4KPj4+Pj4+Pj4gVGhlICJtcWQgcGVy IHVzZXJzcGFjZSIgaXMgZG9uZSBwZXIgdXNlcnNwYWNlIHF1ZXVlIGNyZWF0aW9uLiBIb3dldmVy LCBhcyBJCj4+Pj4+Pj4+IHNhaWQsIGl0IGhhcyBhbiB1cHBlciBsaW1pdCBvZiAxMjhNQiBvbiBL ViwgYW5kIGNvbnNpZGVyaW5nIHRoZSAyRyBsb2NhbAo+Pj4+Pj4+PiBtZW1vcnksIEkgdGhpbmsg aXQgaXMgT0suCj4+Pj4+Pj4+IFRoZSBydW5saXN0cyBhcmUgYWxzbyBkb25lIG9uIHVzZXJzcGFj ZSBxdWV1ZSBjcmVhdGlvbi9kZWxldGlvbiwgYnV0IHdlIG9ubHkKPj4+Pj4+Pj4gaGF2ZSAxIG9y IDIgcnVubGlzdHMgcGVyIGRldmljZSwgc28gaXQgaXMgbm90IHRoYXQgYmFkLgo+Pj4+Pj4+Cj4+ Pj4+Pj4gMkcgbG9jYWwgbWVtb3J5ID8gWW91IGNhbiBub3QgYXNzdW1lIGFueXRoaW5nIG9uIHVz ZXJzaWRlIGNvbmZpZ3VyYXRpb24gc29tZQo+Pj4+Pj4+IG9uZSBtaWdodCBidWlsZCBhbiBoc2Eg Y29tcHV0ZXIgd2l0aCA1MTJNIGFuZCBzdGlsbCBleHBlY3QgYSBmdW5jdGlvbmluZwo+Pj4+Pj4+ IGRlc2t0b3AuCj4+Pj4+PiBGaXJzdCBvZiBhbGwsIEknbSBvbmx5IGNvbnNpZGVyaW5nIEthdmVy aSBjb21wdXRlciwgbm90ICJoc2EiIGNvbXB1dGVyLgo+Pj4+Pj4gU2Vjb25kLCBJIHdvdWxkIGlt YWdpbmUgd2UgY2FuIGJ1aWxkIHNvbWUgcHJvdGVjdGlvbiBhcm91bmQgaXQsIGxpa2UKPj4+Pj4+ IGNoZWNraW5nIHRvdGFsIGxvY2FsIG1lbW9yeSBhbmQgbGltaXQgbnVtYmVyIG9mIHF1ZXVlcyBi YXNlZCBvbiBzb21lCj4+Pj4+PiBwZXJjZW50YWdlIG9mIHRoYXQgdG90YWwgbG9jYWwgbWVtb3J5 LiBTbywgaWYgc29tZW9uZSB3aWxsIGhhdmUgb25seQo+Pj4+Pj4gNTEyTSwgaGUgd2lsbCBiZSBh YmxlIHRvIG9wZW4gbGVzcyBxdWV1ZXMuCj4+Pj4+Pgo+Pj4+Pj4KPj4+Pj4+Pgo+Pj4+Pj4+IEkg bmVlZCB0byBnbyBsb29rIGludG8gd2hhdCBhbGwgdGhpcyBtcWQgaXMgZm9yLCB3aGF0IGl0IGRv ZXMgYW5kIHdoYXQgaXQgaXMKPj4+Pj4+PiBhYm91dC4gQnV0IHBpbm5pbmcgaXMgcmVhbGx5IGJh ZCBhbmQgdGhpcyBpcyBhbiBpc3N1ZSB3aXRoIHVzZXJzcGFjZSBjb21tYW5kCj4+Pj4+Pj4gc2No ZWR1bGluZyBhbiBpc3N1ZSB0aGF0IG9idmlvdXNseSBBTUQgZmFpbHMgdG8gdGFrZSBpbnRvIGFj Y291bnQgaW4gZGVzaWduCj4+Pj4+Pj4gcGhhc2UuCj4+Pj4+PiBNYXliZSwgYnV0IHRoYXQgaXMg dGhlIEgvVyBkZXNpZ24gbm9uLXRoZS1sZXNzLiBXZSBjYW4ndCB2ZXJ5IHdlbGwKPj4+Pj4+IGNo YW5nZSB0aGUgSC9XLgo+Pj4+Pgo+Pj4+PiBZb3UgY2FuIG5vdCBjaGFuZ2UgdGhlIGhhcmR3YXJl IGJ1dCBpdCBpcyBub3QgYW4gZXhjdXNlIHRvIGFsbG93IGJhZCBkZXNpZ24gdG8KPj4+Pj4gc25l YWsgaW4gc29mdHdhcmUgdG8gd29yayBhcm91bmQgdGhhdC4gU28gaSB3b3VsZCByYXRoZXIgcGVu YWxpemUgYmFkIGhhcmR3YXJlCj4+Pj4+IGRlc2lnbiBhbmQgaGF2ZSBjb21tYW5kIHN1Ym1pc3Np b24gaW4gdGhlIGtlcm5lbCwgdW50aWwgQU1EIGZpeCBpdHMgaGFyZHdhcmUgdG8KPj4+Pj4gYWxs b3cgcHJvcGVyIHNjaGVkdWxpbmcgYnkgdGhlIGtlcm5lbCBhbmQgcHJvcGVyIGNvbnRyb2wgYnkg dGhlIGtlcm5lbC4gCj4+Pj4gSSdtIHNvcnJ5IGJ1dCBJIGRvICpub3QqIHRoaW5rIHRoaXMgaXMg YSBiYWQgZGVzaWduLiBTL1cgc2NoZWR1bGluZyBpbgo+Pj4+IHRoZSBrZXJuZWwgY2FuIG5vdCwg SU1PLCBzY2FsZSB3ZWxsIHRvIDEwMEsgcXVldWVzIGFuZCAxMEsgcHJvY2Vzc2VzLgo+Pj4KPj4+ IEkgYW0gbm90IGFkdm9jYXRpbmcgZm9yIGhhdmluZyBrZXJuZWwgZGVjaWRlIGRvd24gdG8gdGhl IHZlcnkgbGFzdCBkZXRhaWxzLiBJIGFtCj4+PiBhZHZvY2F0aW5nIGZvciBrZXJuZWwgYmVpbmcg YWJsZSB0byBwcmVlbXB0IGF0IGFueSB0aW1lIGFuZCBiZSBhYmxlIHRvIGRlY3JlYXNlCj4+PiBv ciBpbmNyZWFzZSB1c2VyIHF1ZXVlIHByaW9yaXR5IHNvIG92ZXJhbGwga2VybmVsIGlzIGluIGNo YXJnZSBvZiByZXNvdXJjZXMKPj4+IG1hbmFnZW1lbnQgYW5kIGl0IGNhbiBoYW5kbGUgcm9ndWUg Y2xpZW50IGluIHByb3BlciBmYXNoaW9uLgo+Pj4KPj4+Pgo+Pj4+PiBCZWNhdXNlIHJlYWxseSB3 aGVyZSB3ZSB3YW50IHRvIGdvIGlzIGhhdmluZyBHUFUgY2xvc2VyIHRvIGEgQ1BVIGluIHRlcm0g b2Ygc2NoZWR1bGluZwo+Pj4+PiBjYXBhY2l0eSBhbmQgb25jZSB3ZSBnZXQgdGhlcmUgd2Ugd2Fu dCB0aGUga2VybmVsIHRvIGFsd2F5cyBiZSBhYmxlIHRvIHRha2Ugb3Zlcgo+Pj4+PiBhbmQgZG8g d2hhdGV2ZXIgaXQgd2FudHMgYmVoaW5kIHByb2Nlc3MgYmFjay4KPj4+PiBXaG8gZG8geW91IHJl ZmVyIHRvIHdoZW4geW91IHNheSAid2UiID8gQUZBSUssIHRoZSBodyBzY2hlZHVsaW5nCj4+Pj4g ZGlyZWN0aW9uIGlzIHdoZXJlIEFNRCBpcyBub3cgYW5kIHdoZXJlIGl0IGlzIGhlYWRpbmcgaW4g dGhlIGZ1dHVyZS4KPj4+PiBUaGF0IGRvZXNuJ3QgcHJlY2x1ZGUgdGhlIG9wdGlvbiB0byBhbGxv dyB0aGUga2VybmVsIHRvIHRha2Ugb3ZlciBhbmQgZG8KPj4+PiB3aGF0IGhlIHdhbnRzLiBJIGFn cmVlIHRoYXQgaW4gS1Ygd2UgaGF2ZSBhIHByb2JsZW0gd2hlcmUgd2UgY2FuJ3QgZG8gYQo+Pj4+ IG1pZC13YXZlIHByZWVtcHRpb24sIHNvIHRoZW9yZXRpY2FsbHksIGEgbG9uZyBydW5uaW5nIGNv bXB1dGUga2VybmVsIGNhbgo+Pj4+IG1ha2UgdGhpbmdzIG1lc3N5LCBidXQgaW4gQ2Fycml6bywg d2Ugd2lsbCBoYXZlIHRoaXMgYWJpbGl0eS4gSGF2aW5nCj4+Pj4gc2FpZCB0aGF0LCBpdCB3aWxs IG9ubHkgYmUgdGhyb3VnaCB0aGUgQ1AgSC9XIHNjaGVkdWxpbmcuIFNvIEFNRCBpcwo+Pj4+IF9u b3RfIGdvaW5nIHRvIGFiYW5kb24gSC9XIHNjaGVkdWxpbmcuIFlvdSBjYW4gZGlzbGlrZSBpdCwg YnV0IHRoaXMgaXMKPj4+PiB0aGUgc2l0dWF0aW9uLgo+Pj4KPj4+IFdlIHdhcyBmb3IgdGhlIG92 ZXJhbGwgTGludXggY29tbXVuaXR5IGJ1dCBtYXliZSBpIHNob3VsZCBub3QgcHJldGVuZCB0byB0 YWxrCj4+PiBmb3IgYW55b25lIGludGVyZXN0ZWQgaW4gaGF2aW5nIGEgY29tbW9uIHN0YW5kYXJk Lgo+Pj4KPj4+IE15IHBvaW50IGlzIHRoYXQgY3VycmVudCBoYXJkd2FyZSBkbyBub3QgaGF2ZSBh cHByb3JpYXRlIGhhcmR3YXJlIHN1cHBvcnQgZm9yCj4+PiBwcmVlbXB0aW9uIGhlbmNlLCBjdXJy ZW50IGhhcmR3YXJlIHNob3VsZCB1c2UgaW9jdGwgdG8gc2NoZWR1bGUgam9iIGFuZCBBTUQKPj4+ IHNob3VsZCB0aGluayBhIGJpdCBtb3JlIG9uIGNvbW1pdGluZyB0byBhIGRlc2lnbiBhbmQgaGFu ZHdhdmluZyBhbnkgaGFyZHdhcmUKPj4+IHNob3J0IGNvbWluZyBhcyBzb21ldGhpbmcgdGhhdCBj YW4gYmUgd29yayBhcm91bmQgaW4gdGhlIHNvZnR3YXJlLiBUaGUgcGlubmluZwo+Pj4gdGhpbmcg aXMgYnJva2VuIGJ5IGRlc2lnbiwgb25seSB3YXkgdG8gd29yayBhcm91bmQgaXQgaXMgdGhyb3Vn aCBrZXJuZWwgY21kCj4+PiBxdWV1ZSBzY2hlZHVsaW5nIHRoYXQncyBhIGZhY3QuCj4+Cj4+Pgo+ Pj4gT25jZSBoYXJkd2FyZSBzdXBwb3J0IHByb3BlciBwcmVlbXB0aW9uIGFuZCBhbGxvd3MgdG8g bW92ZSBhcm91bmQvZXZpY3QgYnVmZmVyCj4+PiB1c2Ugb24gYmVoYWxmIG9mIHVzZXJzcGFjZSBj b21tYW5kIHF1ZXVlIHRoZW4gd2UgY2FuIGFsbG93IHVzZXJzcGFjZSBzY2hlZHVsaW5nCj4+PiBi dXQgdW50aWwgdGhlbiBteSBwZXJzb25uYWwgb3BpbmlvbiBpcyB0aGF0IGl0IHNob3VsZCBub3Qg YmUgYWxsb3dlZCBhbmQgdGhhdAo+Pj4gcGVvcGxlIHdpbGwgaGF2ZSB0byBwYXkgdGhlIGlvY3Rs IHByaWNlIHdoaWNoIGkgcHJvdmVkIHRvIGJlIHNtYWxsLCBiZWNhdXNlCj4+PiByZWFsbHkgaWYg eW91IDEwMEsgcXVldWUgZWFjaCB3aXRoIG9uZSBqb2IsIGkgd291bGQgbm90IGV4cGVjdCB0aGF0 IGFsbCB0aG9zZQo+Pj4gMTAwSyBqb2Igd2lsbCBjb21wbGV0ZSBpbiBsZXNzIHRpbWUgdGhhbiBp dCB0YWtlcyB0byBleGVjdXRlIGFuIGlvY3RsIGllIGJ5Cj4+PiBldmVuIGlmIHlvdSBkbyBub3Qg aGF2ZSB0aGUgaW9jdGwgZGVsYXkgd2hhdCBldmVyIHlvdSBzY2hlZHVsZSB3aWxsIGhhdmUgdG8K Pj4+IHdhaXQgb24gcHJldmlvdXNseSBzdWJtaXRlZCBqb2JzLgo+Pgo+PiBCdXQgSmVyb21lLCB0 aGUgY29yZSBwcm9ibGVtIHN0aWxsIHJlbWFpbnMgaW4gZWZmZWN0LCBldmVuIHdpdGggeW91cgo+ PiBzdWdnZXN0aW9uLiBJZiBhbiBhcHBsaWNhdGlvbiwgZWl0aGVyIHZpYSB1c2Vyc3BhY2UgcXVl dWUgb3IgdmlhIGlvY3RsLAo+PiBzdWJtaXRzIGEgbG9uZy1ydW5uaW5nIGtlcm5lbCwgdGhhbiB0 aGUgQ1BVIGluIGdlbmVyYWwgY2FuJ3Qgc3RvcCB0aGUKPj4gR1BVIGZyb20gcnVubmluZyBpdC4g QW5kIGlmIHRoYXQga2VybmVsIGRvZXMgd2hpbGUoMSk7IHRoYW4gdGhhdCdzIGl0LAo+PiBnYW1l J3Mgb3ZlciwgYW5kIG5vIG1hdHRlciBob3cgeW91IHN1Ym1pdHRlZCB0aGUgd29yay4gU28gSSBk b24ndCByZWFsbHkKPj4gc2VlIHRoZSBiaWcgYWR2YW50YWdlIGluIHlvdXIgcHJvcG9zYWwuIE9u bHkgaW4gQ1ogd2UgY2FuIHN0b3AgdGhpcyB3YXZlCj4+IChieSBDUCBIL1cgc2NoZWR1bGluZyBv bmx5KS4gV2hhdCBhcmUgeW91IHNheWluZyBpcyBiYXNpY2FsbHkgSSB3b24ndAo+PiBhbGxvdyBw ZW9wbGUgdG8gdXNlIGNvbXB1dGUgb24gTGludXggS1Ygc3lzdGVtIGJlY2F1c2UgaXQgX21heV8g Z2V0IHRoZQo+PiBzeXN0ZW0gc3R1Y2suCj4+Cj4+IFNvIGV2ZW4gaWYgSSByZWFsbHkgd2FudGVk IHRvLCBhbmQgSSBtYXkgYWdyZWUgd2l0aCB5b3UgdGhlb3JldGljYWxseSBvbgo+PiB0aGF0LCBJ IGNhbid0IGZ1bGZpbGwgeW91ciBkZXNpcmUgdG8gbWFrZSB0aGUgImtlcm5lbCBiZWluZyBhYmxl IHRvCj4+IHByZWVtcHQgYXQgYW55IHRpbWUgYW5kIGJlIGFibGUgdG8gZGVjcmVhc2Ugb3IgaW5j cmVhc2UgdXNlciBxdWV1ZQo+PiBwcmlvcml0eSBzbyBvdmVyYWxsIGtlcm5lbCBpcyBpbiBjaGFy Z2Ugb2YgcmVzb3VyY2VzIG1hbmFnZW1lbnQgYW5kIGl0Cj4+IGNhbiBoYW5kbGUgcm9ndWUgY2xp ZW50IGluIHByb3BlciBmYXNoaW9uIi4gTm90IGluIEtWLCBhbmQgSSBndWVzcyBub3QKPj4gaW4g Q1ogYXMgd2VsbC4KPj4KPj4gCU9kZWQKPiAKPiBJIGRvIHVuZGVyc3RhbmQgdGhhdCBidXQgdXNp bmcga2VybmVsIGlvY3RsIHByb3ZpZGUgdGhlIHNhbWUga2luZCBvZiBjb250cm9sCj4gYXMgd2Ug aGF2ZSBub3cgaWUgd2UgY2FuIGJpbmQvdW5iaW5kIGJ1ZmZlciBvbiBwZXIgY29tbWFuZCBidWZm ZXIgc3VibWlzc2lvbgo+IGJhc2lzLCBqdXN0IGxpa2Ugd2l0aCBjdXJyZW50IGdyYXBoaWMgb3Ig Y29tcHV0ZSBzdHVmZi4KPiAKPiBZZXMgY3VycmVudCBncmFwaGljIGFuZCBjb21wdXRlIHN0dWZm IGNhbiBsYXVuY2ggYSB3aGlsZSBhbmQgbmV2ZXIgcmV0dXJuIGJhY2sKPiBhbmQgeWVzIGN1cnJl bnRseSB3ZSBoYXZlIG5vdGhpbmcgYWdhaW5zdCB0aGF0IGJ1dCB3ZSBzaG91bGQgYW5kIHNvbHV0 aW9uIHdvdWxkCj4gYmUgc2ltcGxlIGp1c3Qga2lsbCB0aGUgZ3B1IHRocmVhZC4KPiAKT0ssIHNv IGluIHRoYXQgY2FzZSwgdGhlIGtlcm5lbCBjYW4gc2ltcGxlIHVubWFwIGFsbCB0aGUgcXVldWVz IGJ5CnNpbXBseSB3cml0aW5nIGFuIFVOTUFQX1FVRVVFUyBwYWNrZXQgdG8gdGhlIEhJUS4gRXZl biBpZiB0aGUgcXVldWVzIGFyZQp1c2Vyc3BhY2UsIHRoZXkgd2lsbCBub3QgYmUgbWFwcGVkIHRv IHRoZSBpbnRlcm5hbCBDUCBzY2hlZHVsZXIuCkRvZXMgdGhhdCBzYXRpc2Z5IHRoZSBrZXJuZWwg Y29udHJvbCBsZXZlbCB5b3Ugd2FudCA/CgoJT2RlZAo+Pgo+Pj4KPj4+Pj4KPj4+Pj4+Pj4+Cj4+ Pj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pj4+IEl0IG1pZ2h0IGJlIGJldHRlciB0byBhZGQgYSBkcml2ZXJz L2dwdS9kcm0vYW1kIGRpcmVjdG9yeSBhbmQgYWRkIGNvbW1vbgo+Pj4+Pj4+Pj4+PiBzdHVmZiB0 aGVyZS4KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gR2l2ZW4gdGhhdCB0aGlzIGlzIG5vdCBpbnRl bmRlZCB0byBiZSBmaW5hbCBIU0EgYXBpIEFGQUlDVCB0aGVuIGkgd291bGQKPj4+Pj4+Pj4+Pj4g c2F5IHRoaXMgZmFyIGJldHRlciB0byBhdm9pZCB0aGUgd2hvbGUga2ZkIG1vZHVsZSBhbmQgYWRk IGlvY3RsIHRvIHJhZGVvbi4KPj4+Pj4+Pj4+Pj4gVGhpcyB3b3VsZCBhdm9pZCBjcmF6eSBjb21t dW5pY2F0aW9uIGJ0dyByYWRlb24gYW5kIGtmZC4KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gVGhl IHdob2xlIGFwZXJ0dXJlIGJ1c2luZXNzIG5lZWRzIHNvbWUgc2VyaW91cyBleHBsYW5hdGlvbi4g RXNwZWNpYWx5IGFzCj4+Pj4+Pj4+Pj4+IHlvdSB3YW50IHRvIHVzZSB1c2Vyc3BhY2UgYWRkcmVz cyB0aGVyZSBpcyBub3RoaW5nIHRvIHByZXZlbnQgdXNlcnNwYWNlCj4+Pj4+Pj4+Pj4+IHByb2dy YW0gZnJvbSBhbGxvY2F0aW5nIHRoaW5ncyBhdCBhZGRyZXNzIHlvdSByZXNlcnZlIGZvciBsZHMs IHNjcmF0Y2gsCj4+Pj4+Pj4+Pj4+IC4uLiBvbmx5IHNhbmUgd2F5IHdvdWxkIGJlIHRvIG1vdmUg dGhvc2UgbGRzLCBzY3JhdGNoIGluc2lkZSB0aGUgdmlydHVhbAo+Pj4+Pj4+Pj4+PiBhZGRyZXNz IHJlc2VydmVkIGZvciBrZXJuZWwgKHNlZSBrZXJuZWwgbWVtb3J5IG1hcCkuCj4+Pj4+Pj4+Pj4+ Cj4+Pj4+Pj4+Pj4+IFRoZSB3aG9sZSBidXNpbmVzcyBvZiBsb2NraW5nIHBlcmZvcm1hbmNlIGNv dW50ZXIgZm9yIGV4Y2x1c2l2ZSBwZXIgcHJvY2Vzcwo+Pj4+Pj4+Pj4+PiBhY2Nlc3MgaXMgYSBi aWcgTk8uIFdoaWNoIGxlYWRzIG1lIHRvIHRoZSBxdWVzdGlvbmFibGUgdXNlZnVsbG5lc3Mgb2Yg dXNlcgo+Pj4+Pj4+Pj4+PiBzcGFjZSBjb21tYW5kIHJpbmcuCj4+Pj4+Pj4+Pj4gVGhhdCdzIGxp a2Ugc2F5aW5nOiAiV2hpY2ggbGVhZHMgbWUgdG8gdGhlIHF1ZXN0aW9uYWJsZSB1c2VmdWxuZXNz IG9mIEhTQSIuIEkKPj4+Pj4+Pj4+PiBmaW5kIGl0IGFuYWxvZ291cyB0byBhIHNpdHVhdGlvbiB3 aGVyZSBhIG5ldHdvcmsgbWFpbnRhaW5lciBuYWNraW5nIGEgZHJpdmVyCj4+Pj4+Pj4+Pj4gZm9y IGEgbmV0d29yayBjYXJkLCB3aGljaCBpcyBzbG93ZXIgdGhhbiBhIGRpZmZlcmVudCBuZXR3b3Jr IGNhcmQuIERvZXNuJ3QKPj4+Pj4+Pj4+PiBzZWVtIHJlYXNvbmFibGUgdGhpcyBzaXR1YXRpb24g aXMgd291bGQgaGFwcGVuLiBIZSB3b3VsZCBzdGlsbCBwdXQgYm90aCB0aGUKPj4+Pj4+Pj4+PiBk cml2ZXJzIGluIHRoZSBrZXJuZWwgYmVjYXVzZSBwZW9wbGUgd2FudCB0byB1c2UgdGhlIEgvVyBh bmQgaXRzIGZlYXR1cmVzLiBTbywKPj4+Pj4+Pj4+PiBJIGRvbid0IHRoaW5rIHRoaXMgaXMgYSB2 YWxpZCByZWFzb24gdG8gTkFDSyB0aGUgZHJpdmVyLgo+Pj4+Pj4+Cj4+Pj4+Pj4gTGV0IG1lIHJl cGhyYXNlLCBkcm9wIHRoZSB0aGUgcGVyZm9ybWFuY2UgY291bnRlciBpb2N0bCBhbmQgbW9kdWxv IG1lbW9yeSBwaW5uaW5nCj4+Pj4+Pj4gaSBzZWUgbm8gb2JqZWN0aW9uLiBJbiBvdGhlciB3b3Jk LCBpIGFtIG5vdCBOQUNLSU5HIHdob2xlIHBhdGNoc2V0IGkgYW0gTkFDS0lORwo+Pj4+Pj4+IHRo ZSBwZXJmb3JtYW5jZSBpb2N0bC4KPj4+Pj4+Pgo+Pj4+Pj4+IEFnYWluIHRoaXMgaXMgYW5vdGhl ciBhcmd1bWVudCBmb3Igcm91bmQgdHJpcCB0byB0aGUga2VybmVsLiBBcyBpbnNpZGUga2VybmVs IHlvdQo+Pj4+Pj4+IGNvdWxkIHByb3Blcmx5IGRvIGV4Y2x1c2l2ZSBncHUgY291bnRlciBhY2Nl c3MgYWNjcm9zcyBzaW5nbGUgdXNlciBjbWQgYnVmZmVyCj4+Pj4+Pj4gZXhlY3V0aW9uLgo+Pj4+ Pj4+Cj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gSSBvbmx5IHNlZSBpc3N1ZXMgd2l0aCB0aGF0LiBG aXJzdCBhbmQgZm9yZW1vc3QgaSB3b3VsZAo+Pj4+Pj4+Pj4+PiBuZWVkIHRvIHNlZSBzb2xpZCBm aWd1cmVzIHRoYXQga2VybmVsIGlvY3RsIG9yIHN5c2NhbGwgaGFzIGEgaGlnaGVyIGFuCj4+Pj4+ Pj4+Pj4+IG92ZXJoZWFkIHRoYXQgaXMgbWVhc3VyYWJsZSBpbiBhbnkgbWVhbmluZyBmdWxsIHdh eSBhZ2FpbnN0IGEgc2ltcGxlCj4+Pj4+Pj4+Pj4+IGZ1bmN0aW9uIGNhbGwuIEkga25vdyB0aGUg dXNlcnNwYWNlIGNvbW1hbmQgcmluZyBpcyBhIGJpZyBtYXJrZXRpbmcgZmVhdHVyZXMKPj4+Pj4+ Pj4+Pj4gdGhhdCBwbGVhc2UgaWdub3JhbnQgdXNlcnNwYWNlIHByb2dyYW1tZXIuIEJ1dCByZWFs bHkgdGhpcyBvbmx5IGJyaW5ncyBpc3N1ZXMKPj4+Pj4+Pj4+Pj4gYW5kIGZvciBhYnNvbHV0ZWx5 IG5vdCB1cHNpZGUgYWZhaWN0Lgo+Pj4+Pj4+Pj4+IFJlYWxseSA/IFlvdSB0aGluayB0aGF0IGRv aW5nIGEgY29udGV4dCBzd2l0Y2ggdG8ga2VybmVsIHNwYWNlLCB3aXRoIGFsbCBpdHMKPj4+Pj4+ Pj4+PiBvdmVyaGVhZCwgaXMgX25vdF8gbW9yZSBleHBhbnNpdmUgdGhhbiBqdXN0IGNhbGxpbmcg YSBmdW5jdGlvbiBpbiB1c2Vyc3BhY2UKPj4+Pj4+Pj4+PiB3aGljaCBvbmx5IHB1dHMgYSBidWZm ZXIgb24gYSByaW5nIGFuZCB3cml0ZXMgYSBkb29yYmVsbCA/Cj4+Pj4+Pj4KPj4+Pj4+PiBJIGFt IHNheWluZyB0aGUgb3ZlcmhlYWQgaXMgbm90IHRoYXQgYmlnIGFuZCBpdCBwcm9iYWJseSB3aWxs IG5vdCBtYXR0ZXIgaW4gbW9zdAo+Pj4+Pj4+IHVzZWNhc2UuIEZvciBpbnN0YW5jZSBpIGRpZCB3 cm90ZSB0aGUgbW9zdCB1c2VsZXNzIGtlcm5lbCBtb2R1bGUgdGhhdCBhZGQgdHdvCj4+Pj4+Pj4g bnVtYmVyIHRocm91Z2ggYW4gaW9jdGwgKGh0dHA6Ly9wZW9wbGUuZnJlZWRlc2t0b3Aub3JnL35n bGlzc2UvYWRkZXIudGFyKSBhbmQKPj4+Pj4+PiBpdCB0YWtlcyB+MC4zNW1pY3Jvc2Vjb25kcyB3 aXRoIGlvY3RsIHdoaWxlIGZ1bmN0aW9uIGlzIH4wLjAyNW1pY3Jvc2Vjb25kcyBzbwo+Pj4+Pj4+ IGlvY3RsIGlzIDEzIHRpbWVzIHNsb3dlci4KPj4+Pj4+Pgo+Pj4+Pj4+IE5vdyBpZiB0aGVyZSBp cyBlbm91Z2ggZGF0YSB0aGF0IHNob3dzIHRoYXQgYSBzaWduaWZpY2FudCBwZXJjZW50YWdlIG9m IGpvYnMKPj4+Pj4+PiBzdWJtaXRlZCB0byB0aGUgR1BVIHdpbGwgdGFrZSBsZXNzIHRoYXQgMC4z NW1pY3Jvc2Vjb25kIHRoZW4geWVzIHVzZXJzcGFjZQo+Pj4+Pj4+IHNjaGVkdWxpbmcgZG9lcyBt YWtlIHNlbnNlLiBCdXQgc28gZmFyIGFsbCB3ZSBoYXZlIGlzIGhhbmR3YXZpbmcgd2l0aCBubyBk YXRhCj4+Pj4+Pj4gdG8gc3VwcG9ydCBhbnkgZmFjdHMuCj4+Pj4+Pj4KPj4+Pj4+Pgo+Pj4+Pj4+ IE5vdyBpZiB3ZSB3YW50IHRvIHNjaGVkdWxlIGZyb20gdXNlcnNwYWNlIHRoYW4geW91IHdpbGwg bmVlZCB0byBkbyBzb21ldGhpbmcKPj4+Pj4+PiBhYm91dCB0aGUgcGlubmluZywgc29tZXRoaW5n IHRoYXQgZ2l2ZXMgY29udHJvbCB0byBrZXJuZWwgc28gdGhhdCBrZXJuZWwgY2FuCj4+Pj4+Pj4g dW5waW4gd2hlbiBpdCB3YW50cyBhbmQgbW92ZSBvYmplY3Qgd2hlbiBpdCB3YW50cyBubyBtYXR0 ZXIgd2hhdCB1c2Vyc3BhY2UgaXMKPj4+Pj4+PiBkb2luZy4KPj4+Pj4+Pgo+Pj4+Pj4+Pj4+Pgo+ Pj4KPj4+IC0tCj4+PiBUbyB1bnN1YnNjcmliZSwgc2VuZCBhIG1lc3NhZ2Ugd2l0aCAndW5zdWJz Y3JpYmUgbGludXgtbW0nIGluCj4+PiB0aGUgYm9keSB0byBtYWpvcmRvbW9Aa3ZhY2sub3JnLiAg Rm9yIG1vcmUgaW5mbyBvbiBMaW51eCBNTSwKPj4+IHNlZTogaHR0cDovL3d3dy5saW51eC1tbS5v cmcvIC4KPj4+IERvbid0IGVtYWlsOiA8YSBocmVmPW1haWx0bzoiZG9udEBrdmFjay5vcmciPiBl bWFpbEBrdmFjay5vcmcgPC9hPgo+Pj4KPj4KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fCmRyaS1kZXZlbCBtYWlsaW5nIGxpc3QKZHJpLWRldmVsQGxpc3Rz LmZyZWVkZXNrdG9wLm9yZwpodHRwOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlz dGluZm8vZHJpLWRldmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qa0-f53.google.com (mail-qa0-f53.google.com [209.85.216.53]) by kanga.kvack.org (Postfix) with ESMTP id 0DB086B0035 for ; Mon, 21 Jul 2014 17:56:30 -0400 (EDT) Received: by mail-qa0-f53.google.com with SMTP id v10so5701724qac.26 for ; Mon, 21 Jul 2014 14:56:29 -0700 (PDT) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0186.outbound.protection.outlook.com. [207.46.163.186]) by mx.google.com with ESMTPS id j21si28363081qgd.115.2014.07.21.14.56.28 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 21 Jul 2014 14:56:28 -0700 (PDT) Message-ID: <53CD8C7D.9010106@amd.com> Date: Tue, 22 Jul 2014 00:56:13 +0300 From: Oded Gabbay MIME-Version: 1.0 Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver References: <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <53CD1FB6.1000602@amd.com> <20140721155437.GA4519@gmail.com> <53CD5122.5040804@amd.com> <20140721181433.GA5196@gmail.com> <53CD5DBC.7010301@amd.com> <20140721185940.GA5278@gmail.com> <53CD68BF.4020308@amd.com> <20140721192837.GC5278@gmail.com> In-Reply-To: <20140721192837.GC5278@gmail.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Jerome Glisse Cc: Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm , Evgeny Pinchuk , Alexey Skidanov , Andrew Morton On 21/07/14 22:28, Jerome Glisse wrote: > On Mon, Jul 21, 2014 at 10:23:43PM +0300, Oded Gabbay wrote: >> On 21/07/14 21:59, Jerome Glisse wrote: >>> On Mon, Jul 21, 2014 at 09:36:44PM +0300, Oded Gabbay wrote: >>>> On 21/07/14 21:14, Jerome Glisse wrote: >>>>> On Mon, Jul 21, 2014 at 08:42:58PM +0300, Oded Gabbay wrote: >>>>>> On 21/07/14 18:54, Jerome Glisse wrote: >>>>>>> On Mon, Jul 21, 2014 at 05:12:06PM +0300, Oded Gabbay wrote: >>>>>>>> On 21/07/14 16:39, Christian K=C3=B6nig wrote: >>>>>>>>> Am 21.07.2014 14:36, schrieb Oded Gabbay: >>>>>>>>>> On 20/07/14 20:46, Jerome Glisse wrote: >>>>>>>>>>> On Thu, Jul 17, 2014 at 04:57:25PM +0300, Oded Gabbay wrote: >>>>>>>>>>>> Forgot to cc mailing list on cover letter. Sorry. >>>>>>>>>>>> >>>>>>>>>>>> As a continuation to the existing discussion, here is a v2 p= atch series >>>>>>>>>>>> restructured with a cleaner history and no totally-different= -early-versions >>>>>>>>>>>> of the code. >>>>>>>>>>>> >>>>>>>>>>>> Instead of 83 patches, there are now a total of 25 patches, = where 5 of them >>>>>>>>>>>> are modifications to radeon driver and 18 of them include on= ly amdkfd code. >>>>>>>>>>>> There is no code going away or even modified between patches= , only added. >>>>>>>>>>>> >>>>>>>>>>>> The driver was renamed from radeon_kfd to amdkfd and moved t= o reside under >>>>>>>>>>>> drm/radeon/amdkfd. This move was done to emphasize the fact = that this driver >>>>>>>>>>>> is an AMD-only driver at this point. Having said that, we do= foresee a >>>>>>>>>>>> generic hsa framework being implemented in the future and in= that case, we >>>>>>>>>>>> will adjust amdkfd to work within that framework. >>>>>>>>>>>> >>>>>>>>>>>> As the amdkfd driver should support multiple AMD gfx drivers= , we want to >>>>>>>>>>>> keep it as a seperate driver from radeon. Therefore, the amd= kfd code is >>>>>>>>>>>> contained in its own folder. The amdkfd folder was put under= the radeon >>>>>>>>>>>> folder because the only AMD gfx driver in the Linux kernel a= t this point >>>>>>>>>>>> is the radeon driver. Having said that, we will probably nee= d to move it >>>>>>>>>>>> (maybe to be directly under drm) after we integrate with add= itional AMD gfx >>>>>>>>>>>> drivers. >>>>>>>>>>>> >>>>>>>>>>>> For people who like to review using git, the v2 patch set is= located at: >>>>>>>>>>>> http://cgit.freedesktop.org/~gabbayo/linux/log/?h=3Dkfd-next= -3.17-v2 >>>>>>>>>>>> >>>>>>>>>>>> Written by Oded Gabbayh >>>>>>>>>>> >>>>>>>>>>> So quick comments before i finish going over all patches. The= re is many >>>>>>>>>>> things that need more documentation espacialy as of right now= there is >>>>>>>>>>> no userspace i can go look at. >>>>>>>>>> So quick comments on some of your questions but first of all, = thanks for the >>>>>>>>>> time you dedicated to review the code. >>>>>>>>>>> >>>>>>>>>>> There few show stopper, biggest one is gpu memory pinning thi= s is a big >>>>>>>>>>> no, that would need serious arguments for any hope of convinc= ing me on >>>>>>>>>>> that side. >>>>>>>>>> We only do gpu memory pinning for kernel objects. There are no= userspace >>>>>>>>>> objects that are pinned on the gpu memory in our driver. If th= at is the case, >>>>>>>>>> is it still a show stopper ? >>>>>>>>>> >>>>>>>>>> The kernel objects are: >>>>>>>>>> - pipelines (4 per device) >>>>>>>>>> - mqd per hiq (only 1 per device) >>>>>>>>>> - mqd per userspace queue. On KV, we support up to 1K queues p= er process, for >>>>>>>>>> a total of 512K queues. Each mqd is 151 bytes, but the allocat= ion is done in >>>>>>>>>> 256 alignment. So total *possible* memory is 128MB >>>>>>>>>> - kernel queue (only 1 per device) >>>>>>>>>> - fence address for kernel queue >>>>>>>>>> - runlists for the CP (1 or 2 per device) >>>>>>>>> >>>>>>>>> The main questions here are if it's avoid able to pin down the = memory and if the >>>>>>>>> memory is pinned down at driver load, by request from userspace= or by anything >>>>>>>>> else. >>>>>>>>> >>>>>>>>> As far as I can see only the "mqd per userspace queue" might be= a bit >>>>>>>>> questionable, everything else sounds reasonable. >>>>>>>>> >>>>>>>>> Christian. >>>>>>>> >>>>>>>> Most of the pin downs are done on device initialization. >>>>>>>> The "mqd per userspace" is done per userspace queue creation. Ho= wever, as I >>>>>>>> said, it has an upper limit of 128MB on KV, and considering the = 2G local >>>>>>>> memory, I think it is OK. >>>>>>>> The runlists are also done on userspace queue creation/deletion,= but we only >>>>>>>> have 1 or 2 runlists per device, so it is not that bad. >>>>>>> >>>>>>> 2G local memory ? You can not assume anything on userside configu= ration some >>>>>>> one might build an hsa computer with 512M and still expect a func= tioning >>>>>>> desktop. >>>>>> First of all, I'm only considering Kaveri computer, not "hsa" comp= uter. >>>>>> Second, I would imagine we can build some protection around it, li= ke >>>>>> checking total local memory and limit number of queues based on so= me >>>>>> percentage of that total local memory. So, if someone will have on= ly >>>>>> 512M, he will be able to open less queues. >>>>>> >>>>>> >>>>>>> >>>>>>> I need to go look into what all this mqd is for, what it does and= what it is >>>>>>> about. But pinning is really bad and this is an issue with usersp= ace command >>>>>>> scheduling an issue that obviously AMD fails to take into account= in design >>>>>>> phase. >>>>>> Maybe, but that is the H/W design non-the-less. We can't very well >>>>>> change the H/W. >>>>> >>>>> You can not change the hardware but it is not an excuse to allow ba= d design to >>>>> sneak in software to work around that. So i would rather penalize b= ad hardware >>>>> design and have command submission in the kernel, until AMD fix its= hardware to >>>>> allow proper scheduling by the kernel and proper control by the ker= nel.=20 >>>> I'm sorry but I do *not* think this is a bad design. S/W scheduling = in >>>> the kernel can not, IMO, scale well to 100K queues and 10K processes= . >>> >>> I am not advocating for having kernel decide down to the very last de= tails. I am >>> advocating for kernel being able to preempt at any time and be able t= o decrease >>> or increase user queue priority so overall kernel is in charge of res= ources >>> management and it can handle rogue client in proper fashion. >>> >>>> >>>>> Because really where we want to go is having GPU closer to a CPU in= term of scheduling >>>>> capacity and once we get there we want the kernel to always be able= to take over >>>>> and do whatever it wants behind process back. >>>> Who do you refer to when you say "we" ? AFAIK, the hw scheduling >>>> direction is where AMD is now and where it is heading in the future. >>>> That doesn't preclude the option to allow the kernel to take over an= d do >>>> what he wants. I agree that in KV we have a problem where we can't d= o a >>>> mid-wave preemption, so theoretically, a long running compute kernel= can >>>> make things messy, but in Carrizo, we will have this ability. Having >>>> said that, it will only be through the CP H/W scheduling. So AMD is >>>> _not_ going to abandon H/W scheduling. You can dislike it, but this = is >>>> the situation. >>> >>> We was for the overall Linux community but maybe i should not pretend= to talk >>> for anyone interested in having a common standard. >>> >>> My point is that current hardware do not have approriate hardware sup= port for >>> preemption hence, current hardware should use ioctl to schedule job a= nd AMD >>> should think a bit more on commiting to a design and handwaving any h= ardware >>> short coming as something that can be work around in the software. Th= e pinning >>> thing is broken by design, only way to work around it is through kern= el cmd >>> queue scheduling that's a fact. >> >>> >>> Once hardware support proper preemption and allows to move around/evi= ct buffer >>> use on behalf of userspace command queue then we can allow userspace = scheduling >>> but until then my personnal opinion is that it should not be allowed = and that >>> people will have to pay the ioctl price which i proved to be small, b= ecause >>> really if you 100K queue each with one job, i would not expect that a= ll those >>> 100K job will complete in less time than it takes to execute an ioctl= ie by >>> even if you do not have the ioctl delay what ever you schedule will h= ave to >>> wait on previously submited jobs. >> >> But Jerome, the core problem still remains in effect, even with your >> suggestion. If an application, either via userspace queue or via ioctl= , >> submits a long-running kernel, than the CPU in general can't stop the >> GPU from running it. And if that kernel does while(1); than that's it, >> game's over, and no matter how you submitted the work. So I don't real= ly >> see the big advantage in your proposal. Only in CZ we can stop this wa= ve >> (by CP H/W scheduling only). What are you saying is basically I won't >> allow people to use compute on Linux KV system because it _may_ get th= e >> system stuck. >> >> So even if I really wanted to, and I may agree with you theoretically = on >> that, I can't fulfill your desire to make the "kernel being able to >> preempt at any time and be able to decrease or increase user queue >> priority so overall kernel is in charge of resources management and it >> can handle rogue client in proper fashion". Not in KV, and I guess not >> in CZ as well. >> >> Oded >=20 > I do understand that but using kernel ioctl provide the same kind of co= ntrol > as we have now ie we can bind/unbind buffer on per command buffer submi= ssion > basis, just like with current graphic or compute stuff. >=20 > Yes current graphic and compute stuff can launch a while and never retu= rn back > and yes currently we have nothing against that but we should and soluti= on would > be simple just kill the gpu thread. >=20 OK, so in that case, the kernel can simple unmap all the queues by simply writing an UNMAP_QUEUES packet to the HIQ. Even if the queues are userspace, they will not be mapped to the internal CP scheduler. Does that satisfy the kernel control level you want ? Oded >> >>> >>>>> >>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It might be better to add a drivers/gpu/drm/amd directory and= add common >>>>>>>>>>> stuff there. >>>>>>>>>>> >>>>>>>>>>> Given that this is not intended to be final HSA api AFAICT th= en i would >>>>>>>>>>> say this far better to avoid the whole kfd module and add ioc= tl to radeon. >>>>>>>>>>> This would avoid crazy communication btw radeon and kfd. >>>>>>>>>>> >>>>>>>>>>> The whole aperture business needs some serious explanation. E= specialy as >>>>>>>>>>> you want to use userspace address there is nothing to prevent= userspace >>>>>>>>>>> program from allocating things at address you reserve for lds= , scratch, >>>>>>>>>>> ... only sane way would be to move those lds, scratch inside = the virtual >>>>>>>>>>> address reserved for kernel (see kernel memory map). >>>>>>>>>>> >>>>>>>>>>> The whole business of locking performance counter for exclusi= ve per process >>>>>>>>>>> access is a big NO. Which leads me to the questionable useful= lness of user >>>>>>>>>>> space command ring. >>>>>>>>>> That's like saying: "Which leads me to the questionable useful= ness of HSA". I >>>>>>>>>> find it analogous to a situation where a network maintainer na= cking a driver >>>>>>>>>> for a network card, which is slower than a different network c= ard. Doesn't >>>>>>>>>> seem reasonable this situation is would happen. He would still= put both the >>>>>>>>>> drivers in the kernel because people want to use the H/W and i= ts features. So, >>>>>>>>>> I don't think this is a valid reason to NACK the driver. >>>>>>> >>>>>>> Let me rephrase, drop the the performance counter ioctl and modul= o memory pinning >>>>>>> i see no objection. In other word, i am not NACKING whole patchse= t i am NACKING >>>>>>> the performance ioctl. >>>>>>> >>>>>>> Again this is another argument for round trip to the kernel. As i= nside kernel you >>>>>>> could properly do exclusive gpu counter access accross single use= r cmd buffer >>>>>>> execution. >>>>>>> >>>>>>>>>> >>>>>>>>>>> I only see issues with that. First and foremost i would >>>>>>>>>>> need to see solid figures that kernel ioctl or syscall has a = higher an >>>>>>>>>>> overhead that is measurable in any meaning full way against a= simple >>>>>>>>>>> function call. I know the userspace command ring is a big mar= keting features >>>>>>>>>>> that please ignorant userspace programmer. But really this on= ly brings issues >>>>>>>>>>> and for absolutely not upside afaict. >>>>>>>>>> Really ? You think that doing a context switch to kernel space= , with all its >>>>>>>>>> overhead, is _not_ more expansive than just calling a function= in userspace >>>>>>>>>> which only puts a buffer on a ring and writes a doorbell ? >>>>>>> >>>>>>> I am saying the overhead is not that big and it probably will not= matter in most >>>>>>> usecase. For instance i did wrote the most useless kernel module = that add two >>>>>>> number through an ioctl (http://people.freedesktop.org/~glisse/ad= der.tar) and >>>>>>> it takes ~0.35microseconds with ioctl while function is ~0.025mic= roseconds so >>>>>>> ioctl is 13 times slower. >>>>>>> >>>>>>> Now if there is enough data that shows that a significant percent= age of jobs >>>>>>> submited to the GPU will take less that 0.35microsecond then yes = userspace >>>>>>> scheduling does make sense. But so far all we have is handwaving = with no data >>>>>>> to support any facts. >>>>>>> >>>>>>> >>>>>>> Now if we want to schedule from userspace than you will need to d= o something >>>>>>> about the pinning, something that gives control to kernel so that= kernel can >>>>>>> unpin when it wants and move object when it wants no matter what = userspace is >>>>>>> doing. >>>>>>> >>>>>>>>>>> >>> >>> -- >>> To unsubscribe, send a message with 'unsubscribe linux-mm' in >>> the body to majordomo@kvack.org. For more info on Linux MM, >>> see: http://www.linux-mm.org/ . >>> Don't email: email@kvack.org >>> >> -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753987AbaGUV4a (ORCPT ); Mon, 21 Jul 2014 17:56:30 -0400 Received: from mail-bn1blp0189.outbound.protection.outlook.com ([207.46.163.189]:23580 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751052AbaGUV43 convert rfc822-to-8bit (ORCPT ); Mon, 21 Jul 2014 17:56:29 -0400 X-WSS-ID: 0N930XU-08-IWT-02 X-M-MSG: Message-ID: <53CD8C7D.9010106@amd.com> Date: Tue, 22 Jul 2014 00:56:13 +0300 From: Oded Gabbay Organization: AMD User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Jerome Glisse CC: Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm , "Evgeny Pinchuk" , Alexey Skidanov , Andrew Morton Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver References: <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <53CD1FB6.1000602@amd.com> <20140721155437.GA4519@gmail.com> <53CD5122.5040804@amd.com> <20140721181433.GA5196@gmail.com> <53CD5DBC.7010301@amd.com> <20140721185940.GA5278@gmail.com> <53CD68BF.4020308@amd.com> <20140721192837.GC5278@gmail.com> In-Reply-To: <20140721192837.GC5278@gmail.com> Content-Type: text/plain; charset="UTF-8" X-Originating-IP: [10.224.155.167] Content-Transfer-Encoding: 8BIT X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:165.204.84.222;CTRY:US;IPV:NLI;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(6009001)(428002)(51704005)(52254002)(479174003)(51914003)(24454002)(189002)(199002)(110136001)(81342001)(95666004)(36756003)(64126003)(107046002)(76482001)(21056001)(64706001)(79102001)(31966008)(81542001)(50466002)(65956001)(77982001)(105586002)(74662001)(80022001)(20776003)(106466001)(84676001)(74502001)(47776003)(86362001)(65806001)(15202345003)(54356999)(50986999)(87266999)(65816999)(83506001)(44976005)(93886003)(15975445006)(99396002)(76176999)(87936001)(4396001)(85852003)(92726001)(92566001)(83072002)(83322001)(33656002)(102836001)(101416001)(19580405001)(85306003)(97736001)(46102001)(68736004)(19580395003)(23676002);DIR:OUT;SFP:;SCL:1;SRVR:BLUPR02MB036;H:atltwp02.amd.com;FPR:;MLV:sfv;PTR:InfoDomainNonexistent;MX:1;LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0279B3DD0D Authentication-Results: spf=none (sender IP is 165.204.84.222) smtp.mailfrom=Oded.Gabbay@amd.com; X-OriginatorOrg: amd4.onmicrosoft.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 21/07/14 22:28, Jerome Glisse wrote: > On Mon, Jul 21, 2014 at 10:23:43PM +0300, Oded Gabbay wrote: >> On 21/07/14 21:59, Jerome Glisse wrote: >>> On Mon, Jul 21, 2014 at 09:36:44PM +0300, Oded Gabbay wrote: >>>> On 21/07/14 21:14, Jerome Glisse wrote: >>>>> On Mon, Jul 21, 2014 at 08:42:58PM +0300, Oded Gabbay wrote: >>>>>> On 21/07/14 18:54, Jerome Glisse wrote: >>>>>>> On Mon, Jul 21, 2014 at 05:12:06PM +0300, Oded Gabbay wrote: >>>>>>>> On 21/07/14 16:39, Christian König wrote: >>>>>>>>> Am 21.07.2014 14:36, schrieb Oded Gabbay: >>>>>>>>>> On 20/07/14 20:46, Jerome Glisse wrote: >>>>>>>>>>> On Thu, Jul 17, 2014 at 04:57:25PM +0300, Oded Gabbay wrote: >>>>>>>>>>>> Forgot to cc mailing list on cover letter. Sorry. >>>>>>>>>>>> >>>>>>>>>>>> As a continuation to the existing discussion, here is a v2 patch series >>>>>>>>>>>> restructured with a cleaner history and no totally-different-early-versions >>>>>>>>>>>> of the code. >>>>>>>>>>>> >>>>>>>>>>>> Instead of 83 patches, there are now a total of 25 patches, where 5 of them >>>>>>>>>>>> are modifications to radeon driver and 18 of them include only amdkfd code. >>>>>>>>>>>> There is no code going away or even modified between patches, only added. >>>>>>>>>>>> >>>>>>>>>>>> The driver was renamed from radeon_kfd to amdkfd and moved to reside under >>>>>>>>>>>> drm/radeon/amdkfd. This move was done to emphasize the fact that this driver >>>>>>>>>>>> is an AMD-only driver at this point. Having said that, we do foresee a >>>>>>>>>>>> generic hsa framework being implemented in the future and in that case, we >>>>>>>>>>>> will adjust amdkfd to work within that framework. >>>>>>>>>>>> >>>>>>>>>>>> As the amdkfd driver should support multiple AMD gfx drivers, we want to >>>>>>>>>>>> keep it as a seperate driver from radeon. Therefore, the amdkfd code is >>>>>>>>>>>> contained in its own folder. The amdkfd folder was put under the radeon >>>>>>>>>>>> folder because the only AMD gfx driver in the Linux kernel at this point >>>>>>>>>>>> is the radeon driver. Having said that, we will probably need to move it >>>>>>>>>>>> (maybe to be directly under drm) after we integrate with additional AMD gfx >>>>>>>>>>>> drivers. >>>>>>>>>>>> >>>>>>>>>>>> For people who like to review using git, the v2 patch set is located at: >>>>>>>>>>>> http://cgit.freedesktop.org/~gabbayo/linux/log/?h=kfd-next-3.17-v2 >>>>>>>>>>>> >>>>>>>>>>>> Written by Oded Gabbayh >>>>>>>>>>> >>>>>>>>>>> So quick comments before i finish going over all patches. There is many >>>>>>>>>>> things that need more documentation espacialy as of right now there is >>>>>>>>>>> no userspace i can go look at. >>>>>>>>>> So quick comments on some of your questions but first of all, thanks for the >>>>>>>>>> time you dedicated to review the code. >>>>>>>>>>> >>>>>>>>>>> There few show stopper, biggest one is gpu memory pinning this is a big >>>>>>>>>>> no, that would need serious arguments for any hope of convincing me on >>>>>>>>>>> that side. >>>>>>>>>> We only do gpu memory pinning for kernel objects. There are no userspace >>>>>>>>>> objects that are pinned on the gpu memory in our driver. If that is the case, >>>>>>>>>> is it still a show stopper ? >>>>>>>>>> >>>>>>>>>> The kernel objects are: >>>>>>>>>> - pipelines (4 per device) >>>>>>>>>> - mqd per hiq (only 1 per device) >>>>>>>>>> - mqd per userspace queue. On KV, we support up to 1K queues per process, for >>>>>>>>>> a total of 512K queues. Each mqd is 151 bytes, but the allocation is done in >>>>>>>>>> 256 alignment. So total *possible* memory is 128MB >>>>>>>>>> - kernel queue (only 1 per device) >>>>>>>>>> - fence address for kernel queue >>>>>>>>>> - runlists for the CP (1 or 2 per device) >>>>>>>>> >>>>>>>>> The main questions here are if it's avoid able to pin down the memory and if the >>>>>>>>> memory is pinned down at driver load, by request from userspace or by anything >>>>>>>>> else. >>>>>>>>> >>>>>>>>> As far as I can see only the "mqd per userspace queue" might be a bit >>>>>>>>> questionable, everything else sounds reasonable. >>>>>>>>> >>>>>>>>> Christian. >>>>>>>> >>>>>>>> Most of the pin downs are done on device initialization. >>>>>>>> The "mqd per userspace" is done per userspace queue creation. However, as I >>>>>>>> said, it has an upper limit of 128MB on KV, and considering the 2G local >>>>>>>> memory, I think it is OK. >>>>>>>> The runlists are also done on userspace queue creation/deletion, but we only >>>>>>>> have 1 or 2 runlists per device, so it is not that bad. >>>>>>> >>>>>>> 2G local memory ? You can not assume anything on userside configuration some >>>>>>> one might build an hsa computer with 512M and still expect a functioning >>>>>>> desktop. >>>>>> First of all, I'm only considering Kaveri computer, not "hsa" computer. >>>>>> Second, I would imagine we can build some protection around it, like >>>>>> checking total local memory and limit number of queues based on some >>>>>> percentage of that total local memory. So, if someone will have only >>>>>> 512M, he will be able to open less queues. >>>>>> >>>>>> >>>>>>> >>>>>>> I need to go look into what all this mqd is for, what it does and what it is >>>>>>> about. But pinning is really bad and this is an issue with userspace command >>>>>>> scheduling an issue that obviously AMD fails to take into account in design >>>>>>> phase. >>>>>> Maybe, but that is the H/W design non-the-less. We can't very well >>>>>> change the H/W. >>>>> >>>>> You can not change the hardware but it is not an excuse to allow bad design to >>>>> sneak in software to work around that. So i would rather penalize bad hardware >>>>> design and have command submission in the kernel, until AMD fix its hardware to >>>>> allow proper scheduling by the kernel and proper control by the kernel. >>>> I'm sorry but I do *not* think this is a bad design. S/W scheduling in >>>> the kernel can not, IMO, scale well to 100K queues and 10K processes. >>> >>> I am not advocating for having kernel decide down to the very last details. I am >>> advocating for kernel being able to preempt at any time and be able to decrease >>> or increase user queue priority so overall kernel is in charge of resources >>> management and it can handle rogue client in proper fashion. >>> >>>> >>>>> Because really where we want to go is having GPU closer to a CPU in term of scheduling >>>>> capacity and once we get there we want the kernel to always be able to take over >>>>> and do whatever it wants behind process back. >>>> Who do you refer to when you say "we" ? AFAIK, the hw scheduling >>>> direction is where AMD is now and where it is heading in the future. >>>> That doesn't preclude the option to allow the kernel to take over and do >>>> what he wants. I agree that in KV we have a problem where we can't do a >>>> mid-wave preemption, so theoretically, a long running compute kernel can >>>> make things messy, but in Carrizo, we will have this ability. Having >>>> said that, it will only be through the CP H/W scheduling. So AMD is >>>> _not_ going to abandon H/W scheduling. You can dislike it, but this is >>>> the situation. >>> >>> We was for the overall Linux community but maybe i should not pretend to talk >>> for anyone interested in having a common standard. >>> >>> My point is that current hardware do not have approriate hardware support for >>> preemption hence, current hardware should use ioctl to schedule job and AMD >>> should think a bit more on commiting to a design and handwaving any hardware >>> short coming as something that can be work around in the software. The pinning >>> thing is broken by design, only way to work around it is through kernel cmd >>> queue scheduling that's a fact. >> >>> >>> Once hardware support proper preemption and allows to move around/evict buffer >>> use on behalf of userspace command queue then we can allow userspace scheduling >>> but until then my personnal opinion is that it should not be allowed and that >>> people will have to pay the ioctl price which i proved to be small, because >>> really if you 100K queue each with one job, i would not expect that all those >>> 100K job will complete in less time than it takes to execute an ioctl ie by >>> even if you do not have the ioctl delay what ever you schedule will have to >>> wait on previously submited jobs. >> >> But Jerome, the core problem still remains in effect, even with your >> suggestion. If an application, either via userspace queue or via ioctl, >> submits a long-running kernel, than the CPU in general can't stop the >> GPU from running it. And if that kernel does while(1); than that's it, >> game's over, and no matter how you submitted the work. So I don't really >> see the big advantage in your proposal. Only in CZ we can stop this wave >> (by CP H/W scheduling only). What are you saying is basically I won't >> allow people to use compute on Linux KV system because it _may_ get the >> system stuck. >> >> So even if I really wanted to, and I may agree with you theoretically on >> that, I can't fulfill your desire to make the "kernel being able to >> preempt at any time and be able to decrease or increase user queue >> priority so overall kernel is in charge of resources management and it >> can handle rogue client in proper fashion". Not in KV, and I guess not >> in CZ as well. >> >> Oded > > I do understand that but using kernel ioctl provide the same kind of control > as we have now ie we can bind/unbind buffer on per command buffer submission > basis, just like with current graphic or compute stuff. > > Yes current graphic and compute stuff can launch a while and never return back > and yes currently we have nothing against that but we should and solution would > be simple just kill the gpu thread. > OK, so in that case, the kernel can simple unmap all the queues by simply writing an UNMAP_QUEUES packet to the HIQ. Even if the queues are userspace, they will not be mapped to the internal CP scheduler. Does that satisfy the kernel control level you want ? Oded >> >>> >>>>> >>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It might be better to add a drivers/gpu/drm/amd directory and add common >>>>>>>>>>> stuff there. >>>>>>>>>>> >>>>>>>>>>> Given that this is not intended to be final HSA api AFAICT then i would >>>>>>>>>>> say this far better to avoid the whole kfd module and add ioctl to radeon. >>>>>>>>>>> This would avoid crazy communication btw radeon and kfd. >>>>>>>>>>> >>>>>>>>>>> The whole aperture business needs some serious explanation. Especialy as >>>>>>>>>>> you want to use userspace address there is nothing to prevent userspace >>>>>>>>>>> program from allocating things at address you reserve for lds, scratch, >>>>>>>>>>> ... only sane way would be to move those lds, scratch inside the virtual >>>>>>>>>>> address reserved for kernel (see kernel memory map). >>>>>>>>>>> >>>>>>>>>>> The whole business of locking performance counter for exclusive per process >>>>>>>>>>> access is a big NO. Which leads me to the questionable usefullness of user >>>>>>>>>>> space command ring. >>>>>>>>>> That's like saying: "Which leads me to the questionable usefulness of HSA". I >>>>>>>>>> find it analogous to a situation where a network maintainer nacking a driver >>>>>>>>>> for a network card, which is slower than a different network card. Doesn't >>>>>>>>>> seem reasonable this situation is would happen. He would still put both the >>>>>>>>>> drivers in the kernel because people want to use the H/W and its features. So, >>>>>>>>>> I don't think this is a valid reason to NACK the driver. >>>>>>> >>>>>>> Let me rephrase, drop the the performance counter ioctl and modulo memory pinning >>>>>>> i see no objection. In other word, i am not NACKING whole patchset i am NACKING >>>>>>> the performance ioctl. >>>>>>> >>>>>>> Again this is another argument for round trip to the kernel. As inside kernel you >>>>>>> could properly do exclusive gpu counter access accross single user cmd buffer >>>>>>> execution. >>>>>>> >>>>>>>>>> >>>>>>>>>>> I only see issues with that. First and foremost i would >>>>>>>>>>> need to see solid figures that kernel ioctl or syscall has a higher an >>>>>>>>>>> overhead that is measurable in any meaning full way against a simple >>>>>>>>>>> function call. I know the userspace command ring is a big marketing features >>>>>>>>>>> that please ignorant userspace programmer. But really this only brings issues >>>>>>>>>>> and for absolutely not upside afaict. >>>>>>>>>> Really ? You think that doing a context switch to kernel space, with all its >>>>>>>>>> overhead, is _not_ more expansive than just calling a function in userspace >>>>>>>>>> which only puts a buffer on a ring and writes a doorbell ? >>>>>>> >>>>>>> I am saying the overhead is not that big and it probably will not matter in most >>>>>>> usecase. For instance i did wrote the most useless kernel module that add two >>>>>>> number through an ioctl (http://people.freedesktop.org/~glisse/adder.tar) and >>>>>>> it takes ~0.35microseconds with ioctl while function is ~0.025microseconds so >>>>>>> ioctl is 13 times slower. >>>>>>> >>>>>>> Now if there is enough data that shows that a significant percentage of jobs >>>>>>> submited to the GPU will take less that 0.35microsecond then yes userspace >>>>>>> scheduling does make sense. But so far all we have is handwaving with no data >>>>>>> to support any facts. >>>>>>> >>>>>>> >>>>>>> Now if we want to schedule from userspace than you will need to do something >>>>>>> about the pinning, something that gives control to kernel so that kernel can >>>>>>> unpin when it wants and move object when it wants no matter what userspace is >>>>>>> doing. >>>>>>> >>>>>>>>>>> >>> >>> -- >>> To unsubscribe, send a message with 'unsubscribe linux-mm' in >>> the body to majordomo@kvack.org. For more info on Linux MM, >>> see: http://www.linux-mm.org/ . >>> Don't email: email@kvack.org >>> >>