From mboxrd@z Thu Jan 1 00:00:00 1970 From: Oded Gabbay Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver Date: Mon, 21 Jul 2014 21:36:44 +0300 Message-ID: <53CD5DBC.7010301@amd.com> References: <53C7D645.3070607@amd.com> <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <53CD1FB6.1000602@amd.com> <20140721155437.GA4519@gmail.com> <53CD5122.5040804@amd.com> <20140721181433.GA5196@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0186.outbound.protection.outlook.com [207.46.163.186]) by gabe.freedesktop.org (Postfix) with ESMTP id BC13E6E441 for ; Mon, 21 Jul 2014 11:36:56 -0700 (PDT) In-Reply-To: <20140721181433.GA5196@gmail.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Jerome Glisse Cc: Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm , Evgeny Pinchuk , Alexey Skidanov , Andrew Morton List-Id: dri-devel@lists.freedesktop.org T24gMjEvMDcvMTQgMjE6MTQsIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4gT24gTW9uLCBKdWwgMjEs IDIwMTQgYXQgMDg6NDI6NThQTSArMDMwMCwgT2RlZCBHYWJiYXkgd3JvdGU6Cj4+IE9uIDIxLzA3 LzE0IDE4OjU0LCBKZXJvbWUgR2xpc3NlIHdyb3RlOgo+Pj4gT24gTW9uLCBKdWwgMjEsIDIwMTQg YXQgMDU6MTI6MDZQTSArMDMwMCwgT2RlZCBHYWJiYXkgd3JvdGU6Cj4+Pj4gT24gMjEvMDcvMTQg MTY6MzksIENocmlzdGlhbiBLw7ZuaWcgd3JvdGU6Cj4+Pj4+IEFtIDIxLjA3LjIwMTQgMTQ6MzYs IHNjaHJpZWIgT2RlZCBHYWJiYXk6Cj4+Pj4+PiBPbiAyMC8wNy8xNCAyMDo0NiwgSmVyb21lIEds aXNzZSB3cm90ZToKPj4+Pj4+PiBPbiBUaHUsIEp1bCAxNywgMjAxNCBhdCAwNDo1NzoyNVBNICsw MzAwLCBPZGVkIEdhYmJheSB3cm90ZToKPj4+Pj4+Pj4gRm9yZ290IHRvIGNjIG1haWxpbmcgbGlz dCBvbiBjb3ZlciBsZXR0ZXIuIFNvcnJ5Lgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiBBcyBhIGNvbnRpbnVh dGlvbiB0byB0aGUgZXhpc3RpbmcgZGlzY3Vzc2lvbiwgaGVyZSBpcyBhIHYyIHBhdGNoIHNlcmll cwo+Pj4+Pj4+PiByZXN0cnVjdHVyZWQgd2l0aCBhIGNsZWFuZXIgaGlzdG9yeSBhbmQgbm8gdG90 YWxseS1kaWZmZXJlbnQtZWFybHktdmVyc2lvbnMKPj4+Pj4+Pj4gb2YgdGhlIGNvZGUuCj4+Pj4+ Pj4+Cj4+Pj4+Pj4+IEluc3RlYWQgb2YgODMgcGF0Y2hlcywgdGhlcmUgYXJlIG5vdyBhIHRvdGFs IG9mIDI1IHBhdGNoZXMsIHdoZXJlIDUgb2YgdGhlbQo+Pj4+Pj4+PiBhcmUgbW9kaWZpY2F0aW9u cyB0byByYWRlb24gZHJpdmVyIGFuZCAxOCBvZiB0aGVtIGluY2x1ZGUgb25seSBhbWRrZmQgY29k ZS4KPj4+Pj4+Pj4gVGhlcmUgaXMgbm8gY29kZSBnb2luZyBhd2F5IG9yIGV2ZW4gbW9kaWZpZWQg YmV0d2VlbiBwYXRjaGVzLCBvbmx5IGFkZGVkLgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiBUaGUgZHJpdmVy IHdhcyByZW5hbWVkIGZyb20gcmFkZW9uX2tmZCB0byBhbWRrZmQgYW5kIG1vdmVkIHRvIHJlc2lk ZSB1bmRlcgo+Pj4+Pj4+PiBkcm0vcmFkZW9uL2FtZGtmZC4gVGhpcyBtb3ZlIHdhcyBkb25lIHRv IGVtcGhhc2l6ZSB0aGUgZmFjdCB0aGF0IHRoaXMgZHJpdmVyCj4+Pj4+Pj4+IGlzIGFuIEFNRC1v bmx5IGRyaXZlciBhdCB0aGlzIHBvaW50LiBIYXZpbmcgc2FpZCB0aGF0LCB3ZSBkbyBmb3Jlc2Vl IGEKPj4+Pj4+Pj4gZ2VuZXJpYyBoc2EgZnJhbWV3b3JrIGJlaW5nIGltcGxlbWVudGVkIGluIHRo ZSBmdXR1cmUgYW5kIGluIHRoYXQgY2FzZSwgd2UKPj4+Pj4+Pj4gd2lsbCBhZGp1c3QgYW1ka2Zk IHRvIHdvcmsgd2l0aGluIHRoYXQgZnJhbWV3b3JrLgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiBBcyB0aGUg YW1ka2ZkIGRyaXZlciBzaG91bGQgc3VwcG9ydCBtdWx0aXBsZSBBTUQgZ2Z4IGRyaXZlcnMsIHdl IHdhbnQgdG8KPj4+Pj4+Pj4ga2VlcCBpdCBhcyBhIHNlcGVyYXRlIGRyaXZlciBmcm9tIHJhZGVv bi4gVGhlcmVmb3JlLCB0aGUgYW1ka2ZkIGNvZGUgaXMKPj4+Pj4+Pj4gY29udGFpbmVkIGluIGl0 cyBvd24gZm9sZGVyLiBUaGUgYW1ka2ZkIGZvbGRlciB3YXMgcHV0IHVuZGVyIHRoZSByYWRlb24K Pj4+Pj4+Pj4gZm9sZGVyIGJlY2F1c2UgdGhlIG9ubHkgQU1EIGdmeCBkcml2ZXIgaW4gdGhlIExp bnV4IGtlcm5lbCBhdCB0aGlzIHBvaW50Cj4+Pj4+Pj4+IGlzIHRoZSByYWRlb24gZHJpdmVyLiBI YXZpbmcgc2FpZCB0aGF0LCB3ZSB3aWxsIHByb2JhYmx5IG5lZWQgdG8gbW92ZSBpdAo+Pj4+Pj4+ PiAobWF5YmUgdG8gYmUgZGlyZWN0bHkgdW5kZXIgZHJtKSBhZnRlciB3ZSBpbnRlZ3JhdGUgd2l0 aCBhZGRpdGlvbmFsIEFNRCBnZngKPj4+Pj4+Pj4gZHJpdmVycy4KPj4+Pj4+Pj4KPj4+Pj4+Pj4g Rm9yIHBlb3BsZSB3aG8gbGlrZSB0byByZXZpZXcgdXNpbmcgZ2l0LCB0aGUgdjIgcGF0Y2ggc2V0 IGlzIGxvY2F0ZWQgYXQ6Cj4+Pj4+Pj4+IGh0dHA6Ly9jZ2l0LmZyZWVkZXNrdG9wLm9yZy9+Z2Fi YmF5by9saW51eC9sb2cvP2g9a2ZkLW5leHQtMy4xNy12Mgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiBXcml0 dGVuIGJ5IE9kZWQgR2FiYmF5aCA8b2RlZC5nYWJiYXlAYW1kLmNvbT4KPj4+Pj4+Pgo+Pj4+Pj4+ IFNvIHF1aWNrIGNvbW1lbnRzIGJlZm9yZSBpIGZpbmlzaCBnb2luZyBvdmVyIGFsbCBwYXRjaGVz LiBUaGVyZSBpcyBtYW55Cj4+Pj4+Pj4gdGhpbmdzIHRoYXQgbmVlZCBtb3JlIGRvY3VtZW50YXRp b24gZXNwYWNpYWx5IGFzIG9mIHJpZ2h0IG5vdyB0aGVyZSBpcwo+Pj4+Pj4+IG5vIHVzZXJzcGFj ZSBpIGNhbiBnbyBsb29rIGF0Lgo+Pj4+Pj4gU28gcXVpY2sgY29tbWVudHMgb24gc29tZSBvZiB5 b3VyIHF1ZXN0aW9ucyBidXQgZmlyc3Qgb2YgYWxsLCB0aGFua3MgZm9yIHRoZQo+Pj4+Pj4gdGlt ZSB5b3UgZGVkaWNhdGVkIHRvIHJldmlldyB0aGUgY29kZS4KPj4+Pj4+Pgo+Pj4+Pj4+IFRoZXJl IGZldyBzaG93IHN0b3BwZXIsIGJpZ2dlc3Qgb25lIGlzIGdwdSBtZW1vcnkgcGlubmluZyB0aGlz IGlzIGEgYmlnCj4+Pj4+Pj4gbm8sIHRoYXQgd291bGQgbmVlZCBzZXJpb3VzIGFyZ3VtZW50cyBm b3IgYW55IGhvcGUgb2YgY29udmluY2luZyBtZSBvbgo+Pj4+Pj4+IHRoYXQgc2lkZS4KPj4+Pj4+ IFdlIG9ubHkgZG8gZ3B1IG1lbW9yeSBwaW5uaW5nIGZvciBrZXJuZWwgb2JqZWN0cy4gVGhlcmUg YXJlIG5vIHVzZXJzcGFjZQo+Pj4+Pj4gb2JqZWN0cyB0aGF0IGFyZSBwaW5uZWQgb24gdGhlIGdw dSBtZW1vcnkgaW4gb3VyIGRyaXZlci4gSWYgdGhhdCBpcyB0aGUgY2FzZSwKPj4+Pj4+IGlzIGl0 IHN0aWxsIGEgc2hvdyBzdG9wcGVyID8KPj4+Pj4+Cj4+Pj4+PiBUaGUga2VybmVsIG9iamVjdHMg YXJlOgo+Pj4+Pj4gLSBwaXBlbGluZXMgKDQgcGVyIGRldmljZSkKPj4+Pj4+IC0gbXFkIHBlciBo aXEgKG9ubHkgMSBwZXIgZGV2aWNlKQo+Pj4+Pj4gLSBtcWQgcGVyIHVzZXJzcGFjZSBxdWV1ZS4g T24gS1YsIHdlIHN1cHBvcnQgdXAgdG8gMUsgcXVldWVzIHBlciBwcm9jZXNzLCBmb3IKPj4+Pj4+ IGEgdG90YWwgb2YgNTEySyBxdWV1ZXMuIEVhY2ggbXFkIGlzIDE1MSBieXRlcywgYnV0IHRoZSBh bGxvY2F0aW9uIGlzIGRvbmUgaW4KPj4+Pj4+IDI1NiBhbGlnbm1lbnQuIFNvIHRvdGFsICpwb3Nz aWJsZSogbWVtb3J5IGlzIDEyOE1CCj4+Pj4+PiAtIGtlcm5lbCBxdWV1ZSAob25seSAxIHBlciBk ZXZpY2UpCj4+Pj4+PiAtIGZlbmNlIGFkZHJlc3MgZm9yIGtlcm5lbCBxdWV1ZQo+Pj4+Pj4gLSBy dW5saXN0cyBmb3IgdGhlIENQICgxIG9yIDIgcGVyIGRldmljZSkKPj4+Pj4KPj4+Pj4gVGhlIG1h aW4gcXVlc3Rpb25zIGhlcmUgYXJlIGlmIGl0J3MgYXZvaWQgYWJsZSB0byBwaW4gZG93biB0aGUg bWVtb3J5IGFuZCBpZiB0aGUKPj4+Pj4gbWVtb3J5IGlzIHBpbm5lZCBkb3duIGF0IGRyaXZlciBs b2FkLCBieSByZXF1ZXN0IGZyb20gdXNlcnNwYWNlIG9yIGJ5IGFueXRoaW5nCj4+Pj4+IGVsc2Uu Cj4+Pj4+Cj4+Pj4+IEFzIGZhciBhcyBJIGNhbiBzZWUgb25seSB0aGUgIm1xZCBwZXIgdXNlcnNw YWNlIHF1ZXVlIiBtaWdodCBiZSBhIGJpdAo+Pj4+PiBxdWVzdGlvbmFibGUsIGV2ZXJ5dGhpbmcg ZWxzZSBzb3VuZHMgcmVhc29uYWJsZS4KPj4+Pj4KPj4+Pj4gQ2hyaXN0aWFuLgo+Pj4+Cj4+Pj4g TW9zdCBvZiB0aGUgcGluIGRvd25zIGFyZSBkb25lIG9uIGRldmljZSBpbml0aWFsaXphdGlvbi4K Pj4+PiBUaGUgIm1xZCBwZXIgdXNlcnNwYWNlIiBpcyBkb25lIHBlciB1c2Vyc3BhY2UgcXVldWUg Y3JlYXRpb24uIEhvd2V2ZXIsIGFzIEkKPj4+PiBzYWlkLCBpdCBoYXMgYW4gdXBwZXIgbGltaXQg b2YgMTI4TUIgb24gS1YsIGFuZCBjb25zaWRlcmluZyB0aGUgMkcgbG9jYWwKPj4+PiBtZW1vcnks IEkgdGhpbmsgaXQgaXMgT0suCj4+Pj4gVGhlIHJ1bmxpc3RzIGFyZSBhbHNvIGRvbmUgb24gdXNl cnNwYWNlIHF1ZXVlIGNyZWF0aW9uL2RlbGV0aW9uLCBidXQgd2Ugb25seQo+Pj4+IGhhdmUgMSBv ciAyIHJ1bmxpc3RzIHBlciBkZXZpY2UsIHNvIGl0IGlzIG5vdCB0aGF0IGJhZC4KPj4+Cj4+PiAy RyBsb2NhbCBtZW1vcnkgPyBZb3UgY2FuIG5vdCBhc3N1bWUgYW55dGhpbmcgb24gdXNlcnNpZGUg Y29uZmlndXJhdGlvbiBzb21lCj4+PiBvbmUgbWlnaHQgYnVpbGQgYW4gaHNhIGNvbXB1dGVyIHdp dGggNTEyTSBhbmQgc3RpbGwgZXhwZWN0IGEgZnVuY3Rpb25pbmcKPj4+IGRlc2t0b3AuCj4+IEZp cnN0IG9mIGFsbCwgSSdtIG9ubHkgY29uc2lkZXJpbmcgS2F2ZXJpIGNvbXB1dGVyLCBub3QgImhz YSIgY29tcHV0ZXIuCj4+IFNlY29uZCwgSSB3b3VsZCBpbWFnaW5lIHdlIGNhbiBidWlsZCBzb21l IHByb3RlY3Rpb24gYXJvdW5kIGl0LCBsaWtlCj4+IGNoZWNraW5nIHRvdGFsIGxvY2FsIG1lbW9y eSBhbmQgbGltaXQgbnVtYmVyIG9mIHF1ZXVlcyBiYXNlZCBvbiBzb21lCj4+IHBlcmNlbnRhZ2Ug b2YgdGhhdCB0b3RhbCBsb2NhbCBtZW1vcnkuIFNvLCBpZiBzb21lb25lIHdpbGwgaGF2ZSBvbmx5 Cj4+IDUxMk0sIGhlIHdpbGwgYmUgYWJsZSB0byBvcGVuIGxlc3MgcXVldWVzLgo+Pgo+Pgo+Pj4K Pj4+IEkgbmVlZCB0byBnbyBsb29rIGludG8gd2hhdCBhbGwgdGhpcyBtcWQgaXMgZm9yLCB3aGF0 IGl0IGRvZXMgYW5kIHdoYXQgaXQgaXMKPj4+IGFib3V0LiBCdXQgcGlubmluZyBpcyByZWFsbHkg YmFkIGFuZCB0aGlzIGlzIGFuIGlzc3VlIHdpdGggdXNlcnNwYWNlIGNvbW1hbmQKPj4+IHNjaGVk dWxpbmcgYW4gaXNzdWUgdGhhdCBvYnZpb3VzbHkgQU1EIGZhaWxzIHRvIHRha2UgaW50byBhY2Nv dW50IGluIGRlc2lnbgo+Pj4gcGhhc2UuCj4+IE1heWJlLCBidXQgdGhhdCBpcyB0aGUgSC9XIGRl c2lnbiBub24tdGhlLWxlc3MuIFdlIGNhbid0IHZlcnkgd2VsbAo+PiBjaGFuZ2UgdGhlIEgvVy4K PiAKPiBZb3UgY2FuIG5vdCBjaGFuZ2UgdGhlIGhhcmR3YXJlIGJ1dCBpdCBpcyBub3QgYW4gZXhj dXNlIHRvIGFsbG93IGJhZCBkZXNpZ24gdG8KPiBzbmVhayBpbiBzb2Z0d2FyZSB0byB3b3JrIGFy b3VuZCB0aGF0LiBTbyBpIHdvdWxkIHJhdGhlciBwZW5hbGl6ZSBiYWQgaGFyZHdhcmUKPiBkZXNp Z24gYW5kIGhhdmUgY29tbWFuZCBzdWJtaXNzaW9uIGluIHRoZSBrZXJuZWwsIHVudGlsIEFNRCBm aXggaXRzIGhhcmR3YXJlIHRvCj4gYWxsb3cgcHJvcGVyIHNjaGVkdWxpbmcgYnkgdGhlIGtlcm5l bCBhbmQgcHJvcGVyIGNvbnRyb2wgYnkgdGhlIGtlcm5lbC4gCkknbSBzb3JyeSBidXQgSSBkbyAq bm90KiB0aGluayB0aGlzIGlzIGEgYmFkIGRlc2lnbi4gUy9XIHNjaGVkdWxpbmcgaW4KdGhlIGtl cm5lbCBjYW4gbm90LCBJTU8sIHNjYWxlIHdlbGwgdG8gMTAwSyBxdWV1ZXMgYW5kIDEwSyBwcm9j ZXNzZXMuCgo+IEJlY2F1c2UgcmVhbGx5IHdoZXJlIHdlIHdhbnQgdG8gZ28gaXMgaGF2aW5nIEdQ VSBjbG9zZXIgdG8gYSBDUFUgaW4gdGVybSBvZiBzY2hlZHVsaW5nCj4gY2FwYWNpdHkgYW5kIG9u Y2Ugd2UgZ2V0IHRoZXJlIHdlIHdhbnQgdGhlIGtlcm5lbCB0byBhbHdheXMgYmUgYWJsZSB0byB0 YWtlIG92ZXIKPiBhbmQgZG8gd2hhdGV2ZXIgaXQgd2FudHMgYmVoaW5kIHByb2Nlc3MgYmFjay4K V2hvIGRvIHlvdSByZWZlciB0byB3aGVuIHlvdSBzYXkgIndlIiA/IEFGQUlLLCB0aGUgaHcgc2No ZWR1bGluZwpkaXJlY3Rpb24gaXMgd2hlcmUgQU1EIGlzIG5vdyBhbmQgd2hlcmUgaXQgaXMgaGVh ZGluZyBpbiB0aGUgZnV0dXJlLgpUaGF0IGRvZXNuJ3QgcHJlY2x1ZGUgdGhlIG9wdGlvbiB0byBh bGxvdyB0aGUga2VybmVsIHRvIHRha2Ugb3ZlciBhbmQgZG8Kd2hhdCBoZSB3YW50cy4gSSBhZ3Jl ZSB0aGF0IGluIEtWIHdlIGhhdmUgYSBwcm9ibGVtIHdoZXJlIHdlIGNhbid0IGRvIGEKbWlkLXdh dmUgcHJlZW1wdGlvbiwgc28gdGhlb3JldGljYWxseSwgYSBsb25nIHJ1bm5pbmcgY29tcHV0ZSBr ZXJuZWwgY2FuCm1ha2UgdGhpbmdzIG1lc3N5LCBidXQgaW4gQ2Fycml6bywgd2Ugd2lsbCBoYXZl IHRoaXMgYWJpbGl0eS4gSGF2aW5nCnNhaWQgdGhhdCwgaXQgd2lsbCBvbmx5IGJlIHRocm91Z2gg dGhlIENQIEgvVyBzY2hlZHVsaW5nLiBTbyBBTUQgaXMKX25vdF8gZ29pbmcgdG8gYWJhbmRvbiBI L1cgc2NoZWR1bGluZy4gWW91IGNhbiBkaXNsaWtlIGl0LCBidXQgdGhpcyBpcwp0aGUgc2l0dWF0 aW9uLgo+IAo+Pj4+Pgo+Pj4+Pj4+Cj4+Pj4+Pj4gSXQgbWlnaHQgYmUgYmV0dGVyIHRvIGFkZCBh IGRyaXZlcnMvZ3B1L2RybS9hbWQgZGlyZWN0b3J5IGFuZCBhZGQgY29tbW9uCj4+Pj4+Pj4gc3R1 ZmYgdGhlcmUuCj4+Pj4+Pj4KPj4+Pj4+PiBHaXZlbiB0aGF0IHRoaXMgaXMgbm90IGludGVuZGVk IHRvIGJlIGZpbmFsIEhTQSBhcGkgQUZBSUNUIHRoZW4gaSB3b3VsZAo+Pj4+Pj4+IHNheSB0aGlz IGZhciBiZXR0ZXIgdG8gYXZvaWQgdGhlIHdob2xlIGtmZCBtb2R1bGUgYW5kIGFkZCBpb2N0bCB0 byByYWRlb24uCj4+Pj4+Pj4gVGhpcyB3b3VsZCBhdm9pZCBjcmF6eSBjb21tdW5pY2F0aW9uIGJ0 dyByYWRlb24gYW5kIGtmZC4KPj4+Pj4+Pgo+Pj4+Pj4+IFRoZSB3aG9sZSBhcGVydHVyZSBidXNp bmVzcyBuZWVkcyBzb21lIHNlcmlvdXMgZXhwbGFuYXRpb24uIEVzcGVjaWFseSBhcwo+Pj4+Pj4+ IHlvdSB3YW50IHRvIHVzZSB1c2Vyc3BhY2UgYWRkcmVzcyB0aGVyZSBpcyBub3RoaW5nIHRvIHBy ZXZlbnQgdXNlcnNwYWNlCj4+Pj4+Pj4gcHJvZ3JhbSBmcm9tIGFsbG9jYXRpbmcgdGhpbmdzIGF0 IGFkZHJlc3MgeW91IHJlc2VydmUgZm9yIGxkcywgc2NyYXRjaCwKPj4+Pj4+PiAuLi4gb25seSBz YW5lIHdheSB3b3VsZCBiZSB0byBtb3ZlIHRob3NlIGxkcywgc2NyYXRjaCBpbnNpZGUgdGhlIHZp cnR1YWwKPj4+Pj4+PiBhZGRyZXNzIHJlc2VydmVkIGZvciBrZXJuZWwgKHNlZSBrZXJuZWwgbWVt b3J5IG1hcCkuCj4+Pj4+Pj4KPj4+Pj4+PiBUaGUgd2hvbGUgYnVzaW5lc3Mgb2YgbG9ja2luZyBw ZXJmb3JtYW5jZSBjb3VudGVyIGZvciBleGNsdXNpdmUgcGVyIHByb2Nlc3MKPj4+Pj4+PiBhY2Nl c3MgaXMgYSBiaWcgTk8uIFdoaWNoIGxlYWRzIG1lIHRvIHRoZSBxdWVzdGlvbmFibGUgdXNlZnVs bG5lc3Mgb2YgdXNlcgo+Pj4+Pj4+IHNwYWNlIGNvbW1hbmQgcmluZy4KPj4+Pj4+IFRoYXQncyBs aWtlIHNheWluZzogIldoaWNoIGxlYWRzIG1lIHRvIHRoZSBxdWVzdGlvbmFibGUgdXNlZnVsbmVz cyBvZiBIU0EiLiBJCj4+Pj4+PiBmaW5kIGl0IGFuYWxvZ291cyB0byBhIHNpdHVhdGlvbiB3aGVy ZSBhIG5ldHdvcmsgbWFpbnRhaW5lciBuYWNraW5nIGEgZHJpdmVyCj4+Pj4+PiBmb3IgYSBuZXR3 b3JrIGNhcmQsIHdoaWNoIGlzIHNsb3dlciB0aGFuIGEgZGlmZmVyZW50IG5ldHdvcmsgY2FyZC4g RG9lc24ndAo+Pj4+Pj4gc2VlbSByZWFzb25hYmxlIHRoaXMgc2l0dWF0aW9uIGlzIHdvdWxkIGhh cHBlbi4gSGUgd291bGQgc3RpbGwgcHV0IGJvdGggdGhlCj4+Pj4+PiBkcml2ZXJzIGluIHRoZSBr ZXJuZWwgYmVjYXVzZSBwZW9wbGUgd2FudCB0byB1c2UgdGhlIEgvVyBhbmQgaXRzIGZlYXR1cmVz LiBTbywKPj4+Pj4+IEkgZG9uJ3QgdGhpbmsgdGhpcyBpcyBhIHZhbGlkIHJlYXNvbiB0byBOQUNL IHRoZSBkcml2ZXIuCj4+Pgo+Pj4gTGV0IG1lIHJlcGhyYXNlLCBkcm9wIHRoZSB0aGUgcGVyZm9y bWFuY2UgY291bnRlciBpb2N0bCBhbmQgbW9kdWxvIG1lbW9yeSBwaW5uaW5nCj4+PiBpIHNlZSBu byBvYmplY3Rpb24uIEluIG90aGVyIHdvcmQsIGkgYW0gbm90IE5BQ0tJTkcgd2hvbGUgcGF0Y2hz ZXQgaSBhbSBOQUNLSU5HCj4+PiB0aGUgcGVyZm9ybWFuY2UgaW9jdGwuCj4+Pgo+Pj4gQWdhaW4g dGhpcyBpcyBhbm90aGVyIGFyZ3VtZW50IGZvciByb3VuZCB0cmlwIHRvIHRoZSBrZXJuZWwuIEFz IGluc2lkZSBrZXJuZWwgeW91Cj4+PiBjb3VsZCBwcm9wZXJseSBkbyBleGNsdXNpdmUgZ3B1IGNv dW50ZXIgYWNjZXNzIGFjY3Jvc3Mgc2luZ2xlIHVzZXIgY21kIGJ1ZmZlcgo+Pj4gZXhlY3V0aW9u Lgo+Pj4KPj4+Pj4+Cj4+Pj4+Pj4gSSBvbmx5IHNlZSBpc3N1ZXMgd2l0aCB0aGF0LiBGaXJzdCBh bmQgZm9yZW1vc3QgaSB3b3VsZAo+Pj4+Pj4+IG5lZWQgdG8gc2VlIHNvbGlkIGZpZ3VyZXMgdGhh dCBrZXJuZWwgaW9jdGwgb3Igc3lzY2FsbCBoYXMgYSBoaWdoZXIgYW4KPj4+Pj4+PiBvdmVyaGVh ZCB0aGF0IGlzIG1lYXN1cmFibGUgaW4gYW55IG1lYW5pbmcgZnVsbCB3YXkgYWdhaW5zdCBhIHNp bXBsZQo+Pj4+Pj4+IGZ1bmN0aW9uIGNhbGwuIEkga25vdyB0aGUgdXNlcnNwYWNlIGNvbW1hbmQg cmluZyBpcyBhIGJpZyBtYXJrZXRpbmcgZmVhdHVyZXMKPj4+Pj4+PiB0aGF0IHBsZWFzZSBpZ25v cmFudCB1c2Vyc3BhY2UgcHJvZ3JhbW1lci4gQnV0IHJlYWxseSB0aGlzIG9ubHkgYnJpbmdzIGlz c3Vlcwo+Pj4+Pj4+IGFuZCBmb3IgYWJzb2x1dGVseSBub3QgdXBzaWRlIGFmYWljdC4KPj4+Pj4+ IFJlYWxseSA/IFlvdSB0aGluayB0aGF0IGRvaW5nIGEgY29udGV4dCBzd2l0Y2ggdG8ga2VybmVs IHNwYWNlLCB3aXRoIGFsbCBpdHMKPj4+Pj4+IG92ZXJoZWFkLCBpcyBfbm90XyBtb3JlIGV4cGFu c2l2ZSB0aGFuIGp1c3QgY2FsbGluZyBhIGZ1bmN0aW9uIGluIHVzZXJzcGFjZQo+Pj4+Pj4gd2hp Y2ggb25seSBwdXRzIGEgYnVmZmVyIG9uIGEgcmluZyBhbmQgd3JpdGVzIGEgZG9vcmJlbGwgPwo+ Pj4KPj4+IEkgYW0gc2F5aW5nIHRoZSBvdmVyaGVhZCBpcyBub3QgdGhhdCBiaWcgYW5kIGl0IHBy b2JhYmx5IHdpbGwgbm90IG1hdHRlciBpbiBtb3N0Cj4+PiB1c2VjYXNlLiBGb3IgaW5zdGFuY2Ug aSBkaWQgd3JvdGUgdGhlIG1vc3QgdXNlbGVzcyBrZXJuZWwgbW9kdWxlIHRoYXQgYWRkIHR3bwo+ Pj4gbnVtYmVyIHRocm91Z2ggYW4gaW9jdGwgKGh0dHA6Ly9wZW9wbGUuZnJlZWRlc2t0b3Aub3Jn L35nbGlzc2UvYWRkZXIudGFyKSBhbmQKPj4+IGl0IHRha2VzIH4wLjM1bWljcm9zZWNvbmRzIHdp dGggaW9jdGwgd2hpbGUgZnVuY3Rpb24gaXMgfjAuMDI1bWljcm9zZWNvbmRzIHNvCj4+PiBpb2N0 bCBpcyAxMyB0aW1lcyBzbG93ZXIuCj4+Pgo+Pj4gTm93IGlmIHRoZXJlIGlzIGVub3VnaCBkYXRh IHRoYXQgc2hvd3MgdGhhdCBhIHNpZ25pZmljYW50IHBlcmNlbnRhZ2Ugb2Ygam9icwo+Pj4gc3Vi bWl0ZWQgdG8gdGhlIEdQVSB3aWxsIHRha2UgbGVzcyB0aGF0IDAuMzVtaWNyb3NlY29uZCB0aGVu IHllcyB1c2Vyc3BhY2UKPj4+IHNjaGVkdWxpbmcgZG9lcyBtYWtlIHNlbnNlLiBCdXQgc28gZmFy IGFsbCB3ZSBoYXZlIGlzIGhhbmR3YXZpbmcgd2l0aCBubyBkYXRhCj4+PiB0byBzdXBwb3J0IGFu eSBmYWN0cy4KPj4+Cj4+Pgo+Pj4gTm93IGlmIHdlIHdhbnQgdG8gc2NoZWR1bGUgZnJvbSB1c2Vy c3BhY2UgdGhhbiB5b3Ugd2lsbCBuZWVkIHRvIGRvIHNvbWV0aGluZwo+Pj4gYWJvdXQgdGhlIHBp bm5pbmcsIHNvbWV0aGluZyB0aGF0IGdpdmVzIGNvbnRyb2wgdG8ga2VybmVsIHNvIHRoYXQga2Vy bmVsIGNhbgo+Pj4gdW5waW4gd2hlbiBpdCB3YW50cyBhbmQgbW92ZSBvYmplY3Qgd2hlbiBpdCB3 YW50cyBubyBtYXR0ZXIgd2hhdCB1c2Vyc3BhY2UgaXMKPj4+IGRvaW5nLgo+Pj4KPj4+Pj4+Pgo+ Pj4+Pj4+IFNvIGkgd291bGQgcmF0aGVyIHNlZSBhIHZlcnkgc2ltcGxlIGlvY3RsIHRoYXQgd3Jp dGUgdGhlIGRvb3JiZWxsIGFuZCBtaWdodAo+Pj4+Pj4+IGRvIG1vcmUgdGhhbiB0aGF0IGluIGNh c2Ugb2YgcmluZy9xdWV1ZSBvdmVyY29tbWl0IHdoZXJlIGl0IHdvdWxkIGZpcnN0IGhhdmUKPj4+ Pj4+PiB0byB3YWl0IGZvciBhIGZyZWUgcmluZy9xdWV1ZSB0byBzY2hlZHVsZSBzdHVmZi4gVGhp cyB3b3VsZCBhbHNvIGFsbG93IHNhbmUKPj4+Pj4+PiBpbXBsZW1lbnRhdGlvbiBvZiB0aGluZ3Mg bGlrZSBwZXJmb3JtYW5jZSBjb3VudGVyIHRoYXQgY291bGQgYmUgYWNxdWlyZSBieQo+Pj4+Pj4+ IGtlcm5lbCBmb3IgZHVyYXRpb24gb2YgYSBqb2Igc3VibWl0dGVkIGJ5IHVzZXJzcGFjZS4gV2hp bGUgc3RpbGwgbm90IG9wdGltYWwKPj4+Pj4+PiB0aGlzIHdvdWxkIGJlIGJldHRlciB0aGF0IHVz ZXJzcGFjZSBsb2NraW5nLgo+Pj4+Pj4+Cj4+Pj4+Pj4KPj4+Pj4+PiBJIG1pZ2h0IGhhdmUgbW9y ZSB0aG91Z2h0cyBvbmNlIGkgYW0gZG9uZSB3aXRoIGFsbCB0aGUgcGF0Y2hlcy4KPj4+Pj4+Pgo+ Pj4+Pj4+IENoZWVycywKPj4+Pj4+PiBKw6lyw7RtZQo+Pj4+Pj4+Cj4+Pj4+Pj4+Cj4+Pj4+Pj4+ IE9yaWdpbmFsIENvdmVyIExldHRlcjoKPj4+Pj4+Pj4KPj4+Pj4+Pj4gVGhpcyBwYXRjaCBzZXQg aW1wbGVtZW50cyBhIEhldGVyb2dlbmVvdXMgU3lzdGVtIEFyY2hpdGVjdHVyZSAoSFNBKSBkcml2 ZXIKPj4+Pj4+Pj4gZm9yIHJhZGVvbi1mYW1pbHkgR1BVcy4KPj4+Pj4+Pj4gSFNBIGFsbG93cyBk aWZmZXJlbnQgcHJvY2Vzc29yIHR5cGVzIChDUFVzLCBEU1BzLCBHUFVzLCBldGMuLikgdG8gc2hh cmUKPj4+Pj4+Pj4gc3lzdGVtIHJlc291cmNlcyBtb3JlIGVmZmVjdGl2ZWx5IHZpYSBIVyBmZWF0 dXJlcyBpbmNsdWRpbmcgc2hhcmVkIHBhZ2VhYmxlCj4+Pj4+Pj4+IG1lbW9yeSwgdXNlcnNwYWNl LWFjY2Vzc2libGUgd29yayBxdWV1ZXMsIGFuZCBwbGF0Zm9ybS1sZXZlbCBhdG9taWNzLiBJbgo+ Pj4+Pj4+PiBhZGRpdGlvbiB0byB0aGUgbWVtb3J5IHByb3RlY3Rpb24gbWVjaGFuaXNtcyBpbiBH UFVWTSBhbmQgSU9NTVV2MiwgdGhlIFNlYQo+Pj4+Pj4+PiBJc2xhbmRzIGZhbWlseSBvZiBHUFVz IGFsc28gcGVyZm9ybXMgSFctbGV2ZWwgdmFsaWRhdGlvbiBvZiBjb21tYW5kcyBwYXNzZWQKPj4+ Pj4+Pj4gaW4gdGhyb3VnaCB0aGUgcXVldWVzIChha2EgcmluZ3MpLgo+Pj4+Pj4+Pgo+Pj4+Pj4+ PiBUaGUgY29kZSBpbiB0aGlzIHBhdGNoIHNldCBpcyBpbnRlbmRlZCB0byBzZXJ2ZSBib3RoIGFz IGEgc2FtcGxlIGRyaXZlciBmb3IKPj4+Pj4+Pj4gb3RoZXIgSFNBLWNvbXBhdGlibGUgaGFyZHdh cmUgZGV2aWNlcyBhbmQgYXMgYSBwcm9kdWN0aW9uIGRyaXZlciBmb3IKPj4+Pj4+Pj4gcmFkZW9u LWZhbWlseSBwcm9jZXNzb3JzLiBUaGUgY29kZSBpcyBhcmNoaXRlY3RlZCB0byBzdXBwb3J0IG11 bHRpcGxlIENQVXMKPj4+Pj4+Pj4gZWFjaCB3aXRoIGNvbm5lY3RlZCBHUFVzLCBhbHRob3VnaCB0 aGUgY3VycmVudCBpbXBsZW1lbnRhdGlvbiBmb2N1c2VzIG9uIGEKPj4+Pj4+Pj4gc2luZ2xlIEth dmVyaS9CZXJsaW4gQVBVLCBhbmQgd29ya3MgYWxvbmdzaWRlIHRoZSBleGlzdGluZyByYWRlb24g a2VybmVsCj4+Pj4+Pj4+IGdyYXBoaWNzIGRyaXZlciAoa2dkKS4KPj4+Pj4+Pj4gQU1EIEdQVXMg ZGVzaWduZWQgZm9yIHVzZSB3aXRoIEhTQSAoU2VhIElzbGFuZHMgYW5kIHVwKSBzaGFyZSBzb21l IGhhcmR3YXJlCj4+Pj4+Pj4+IGZ1bmN0aW9uYWxpdHkgYmV0d2VlbiBIU0EgY29tcHV0ZSBhbmQg cmVndWxhciBnZngvY29tcHV0ZSAobWVtb3J5LAo+Pj4+Pj4+PiBpbnRlcnJ1cHRzLCByZWdpc3Rl cnMpLCB3aGlsZSBvdGhlciBmdW5jdGlvbmFsaXR5IGhhcyBiZWVuIGFkZGVkCj4+Pj4+Pj4+IHNw ZWNpZmljYWxseSBmb3IgSFNBIGNvbXB1dGUgIChodyBzY2hlZHVsZXIgZm9yIHZpcnR1YWxpemVk IGNvbXB1dGUgcmluZ3MpLgo+Pj4+Pj4+PiBBbGwgc2hhcmVkIGhhcmR3YXJlIGlzIG93bmVkIGJ5 IHRoZSByYWRlb24gZ3JhcGhpY3MgZHJpdmVyLCBhbmQgYW4gaW50ZXJmYWNlCj4+Pj4+Pj4+IGJl dHdlZW4ga2ZkIGFuZCBrZ2QgYWxsb3dzIHRoZSBrZmQgdG8gbWFrZSB1c2Ugb2YgdGhvc2Ugc2hh cmVkIHJlc291cmNlcywKPj4+Pj4+Pj4gd2hpbGUgSFNBLXNwZWNpZmljIGZ1bmN0aW9uYWxpdHkg aXMgbWFuYWdlZCBkaXJlY3RseSBieSBrZmQgYnkgc3VibWl0dGluZwo+Pj4+Pj4+PiBwYWNrZXRz IGludG8gYW4gSFNBLXNwZWNpZmljIGNvbW1hbmQgcXVldWUgKHRoZSAiSElRIikuCj4+Pj4+Pj4+ Cj4+Pj4+Pj4+IER1cmluZyBrZmQgbW9kdWxlIGluaXRpYWxpemF0aW9uIGEgY2hhciBkZXZpY2Ug bm9kZSAoL2Rldi9rZmQpIGlzIGNyZWF0ZWQKPj4+Pj4+Pj4gKHN1cnZpdmluZyB1bnRpbCBtb2R1 bGUgZXhpdCksIHdpdGggaW9jdGxzIGZvciBxdWV1ZSBjcmVhdGlvbiAmIG1hbmFnZW1lbnQsCj4+ Pj4+Pj4+IGFuZCBkYXRhIHN0cnVjdHVyZXMgYXJlIGluaXRpYWxpemVkIGZvciBtYW5hZ2luZyBI U0EgZGV2aWNlIHRvcG9sb2d5Lgo+Pj4+Pj4+PiBUaGUgcmVzdCBvZiB0aGUgaW5pdGlhbGl6YXRp b24gaXMgZHJpdmVuIGJ5IGNhbGxzIGZyb20gdGhlIHJhZGVvbiBrZ2QgYXQgdGhlCj4+Pj4+Pj4+ IGZvbGxvd2luZyBwb2ludHMgOgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiAtIHJhZGVvbl9pbml0IChrZmRf aW5pdCkKPj4+Pj4+Pj4gLSByYWRlb25fZXhpdCAoa2ZkX2ZpbmkpCj4+Pj4+Pj4+IC0gcmFkZW9u X2RyaXZlcl9sb2FkX2ttcyAoa2ZkX2RldmljZV9wcm9iZSwga2ZkX2RldmljZV9pbml0KQo+Pj4+ Pj4+PiAtIHJhZGVvbl9kcml2ZXJfdW5sb2FkX2ttcyAoa2ZkX2RldmljZV9maW5pKQo+Pj4+Pj4+ Pgo+Pj4+Pj4+PiBEdXJpbmcgdGhlIHByb2JlIGFuZCBpbml0IHByb2Nlc3NpbmcgcGVyLWRldmlj ZSBkYXRhIHN0cnVjdHVyZXMgYXJlCj4+Pj4+Pj4+IGVzdGFibGlzaGVkIHdoaWNoIGNvbm5lY3Qg dG8gdGhlIGFzc29jaWF0ZWQgZ3JhcGhpY3Mga2VybmVsIGRyaXZlci4gVGhpcwo+Pj4+Pj4+PiBp bmZvcm1hdGlvbiBpcyBleHBvc2VkIHRvIHVzZXJzcGFjZSB2aWEgc3lzZnMsIGFsb25nIHdpdGgg YSB2ZXJzaW9uIG51bWJlcgo+Pj4+Pj4+PiBhbGxvd2luZyB1c2Vyc3BhY2UgdG8gZGV0ZXJtaW5l IGlmIGEgdG9wb2xvZ3kgY2hhbmdlIGhhcyBvY2N1cnJlZCB3aGlsZSBpdAo+Pj4+Pj4+PiB3YXMg cmVhZGluZyBmcm9tIHN5c2ZzLgo+Pj4+Pj4+PiBUaGUgaW50ZXJmYWNlIGJldHdlZW4ga2ZkIGFu ZCBrZ2QgYWxzbyBhbGxvd3MgdGhlIGtmZCB0byByZXF1ZXN0IGJ1ZmZlcgo+Pj4+Pj4+PiBtYW5h Z2VtZW50IHNlcnZpY2VzIGZyb20ga2dkLCBhbmQgYWxsb3dzIGtnZCB0byByb3V0ZSBpbnRlcnJ1 cHQgcmVxdWVzdHMgdG8KPj4+Pj4+Pj4ga2ZkIGNvZGUgc2luY2UgdGhlIGludGVycnVwdCBibG9j ayBpcyBzaGFyZWQgYmV0d2VlbiByZWd1bGFyCj4+Pj4+Pj4+IGdyYXBoaWNzL2NvbXB1dGUgYW5k IEhTQSBjb21wdXRlIHN1YnN5c3RlbXMgaW4gdGhlIEdQVS4KPj4+Pj4+Pj4KPj4+Pj4+Pj4gVGhl IGtmZCBjb2RlIHdvcmtzIHdpdGggYW4gb3BlbiBzb3VyY2UgdXNlcm1vZGUgbGlicmFyeSAoImxp YmhzYWttdCIpIHdoaWNoCj4+Pj4+Pj4+IGlzIGluIHRoZSBmaW5hbCBzdGFnZXMgb2YgSVAgcmV2 aWV3IGFuZCBzaG91bGQgYmUgcHVibGlzaGVkIGluIGEgc2VwYXJhdGUKPj4+Pj4+Pj4gcmVwbyBv dmVyIHRoZSBuZXh0IGZldyBkYXlzLgo+Pj4+Pj4+PiBUaGUgY29kZSBvcGVyYXRlcyBpbiBvbmUg b2YgdGhyZWUgbW9kZXMsIHNlbGVjdGFibGUgdmlhIHRoZSBzY2hlZF9wb2xpY3kKPj4+Pj4+Pj4g bW9kdWxlIHBhcmFtZXRlciA6Cj4+Pj4+Pj4+Cj4+Pj4+Pj4+IC0gc2NoZWRfcG9saWN5PTAgdXNl cyBhIGhhcmR3YXJlIHNjaGVkdWxlciBydW5uaW5nIGluIHRoZSBNRUMgYmxvY2sgd2l0aGluCj4+ Pj4+Pj4+IENQLCBhbmQgYWxsb3dzIG92ZXJzdWJzY3JpcHRpb24gKG1vcmUgcXVldWVzIHRoYW4g SFcgc2xvdHMpCj4+Pj4+Pj4+IC0gc2NoZWRfcG9saWN5PTEgYWxzbyB1c2VzIEhXIHNjaGVkdWxp bmcgYnV0IGRvZXMgbm90IGFsbG93Cj4+Pj4+Pj4+IG92ZXJzdWJzY3JpcHRpb24sIHNvIGNyZWF0 ZV9xdWV1ZSByZXF1ZXN0cyBmYWlsIHdoZW4gd2UgcnVuIG91dCBvZiBIVyBzbG90cwo+Pj4+Pj4+ PiAtIHNjaGVkX3BvbGljeT0yIGRvZXMgbm90IHVzZSBIVyBzY2hlZHVsaW5nLCBzbyB0aGUgZHJp dmVyIG1hbnVhbGx5IGFzc2lnbnMKPj4+Pj4+Pj4gcXVldWVzIHRvIEhXIHNsb3RzIGJ5IHByb2dy YW1taW5nIHJlZ2lzdGVycwo+Pj4+Pj4+Pgo+Pj4+Pj4+PiBUaGUgIm5vIEhXIHNjaGVkdWxpbmci IG9wdGlvbiBpcyBmb3IgZGVidWcgJiBuZXcgaGFyZHdhcmUgYnJpbmd1cCBvbmx5LCBzbwo+Pj4+ Pj4+PiBoYXMgbGVzcyB0ZXN0IGNvdmVyYWdlIHRoYW4gdGhlIG90aGVyIG9wdGlvbnMuIERlZmF1 bHQgaW4gdGhlIGN1cnJlbnQgY29kZQo+Pj4+Pj4+PiBpcyAiSFcgc2NoZWR1bGluZyB3aXRob3V0 IG92ZXJzdWJzY3JpcHRpb24iIHNpbmNlIHRoYXQgaXMgd2hlcmUgd2UgaGF2ZSB0aGUKPj4+Pj4+ Pj4gbW9zdCB0ZXN0IGNvdmVyYWdlIGJ1dCB3ZSBleHBlY3QgdG8gY2hhbmdlIHRoZSBkZWZhdWx0 IHRvICJIVyBzY2hlZHVsaW5nCj4+Pj4+Pj4+IHdpdGggb3ZlcnN1YnNjcmlwdGlvbiIgYWZ0ZXIg ZnVydGhlciB0ZXN0aW5nLiBUaGlzIGVmZmVjdGl2ZWx5IHJlbW92ZXMgdGhlCj4+Pj4+Pj4+IEhX IGxpbWl0IG9uIHRoZSBudW1iZXIgb2Ygd29yayBxdWV1ZXMgYXZhaWxhYmxlIHRvIGFwcGxpY2F0 aW9ucy4KPj4+Pj4+Pj4KPj4+Pj4+Pj4gUHJvZ3JhbXMgcnVubmluZyBvbiB0aGUgR1BVIGFyZSBh c3NvY2lhdGVkIHdpdGggYW4gYWRkcmVzcyBzcGFjZSB0aHJvdWdoIHRoZQo+Pj4+Pj4+PiBWTUlE IGZpZWxkLCB3aGljaCBpcyB0cmFuc2xhdGVkIHRvIGEgdW5pcXVlIFBBU0lEIGF0IGFjY2VzcyB0 aW1lIHZpYSBhIHNldAo+Pj4+Pj4+PiBvZiAxNiBWTUlELXRvLVBBU0lEIG1hcHBpbmcgcmVnaXN0 ZXJzLiBUaGUgYXZhaWxhYmxlIFZNSURzIChjdXJyZW50bHkgMTYpCj4+Pj4+Pj4+IGFyZSBwYXJ0 aXRpb25lZCAodW5kZXIgY29udHJvbCBvZiB0aGUgcmFkZW9uIGtnZCkgYmV0d2VlbiBjdXJyZW50 Cj4+Pj4+Pj4+IGdmeC9jb21wdXRlIGFuZCBIU0EgY29tcHV0ZSwgd2l0aCBlYWNoIGdldHRpbmcg OCBpbiB0aGUgY3VycmVudCBjb2RlLiBUaGUKPj4+Pj4+Pj4gVk1JRC10by1QQVNJRCBtYXBwaW5n IHJlZ2lzdGVycyBhcmUgdXBkYXRlZCBieSB0aGUgSFcgc2NoZWR1bGVyIHdoZW4gdXNlZCwKPj4+ Pj4+Pj4gYW5kIGJ5IGRyaXZlciBjb2RlIGlmIEhXIHNjaGVkdWxpbmcgaXMgbm90IGJlaW5nIHVz ZWQuCj4+Pj4+Pj4+IFRoZSBTZWEgSXNsYW5kcyBjb21wdXRlIHF1ZXVlcyB1c2UgYSBuZXcgImRv b3JiZWxsIiBtZWNoYW5pc20gaW5zdGVhZCBvZiB0aGUKPj4+Pj4+Pj4gZWFybGllciBrZXJuZWwt bWFuYWdlZCB3cml0ZSBwb2ludGVyIHJlZ2lzdGVycy4gRG9vcmJlbGxzIHVzZSBhIHNlcGFyYXRl IEJBUgo+Pj4+Pj4+PiBkZWRpY2F0ZWQgZm9yIHRoaXMgcHVycG9zZSwgYW5kIHBhZ2VzIHdpdGhp biB0aGUgZG9vcmJlbGwgYXBlcnR1cmUgYXJlCj4+Pj4+Pj4+IG1hcHBlZCB0byB1c2Vyc3BhY2Ug KGVhY2ggcGFnZSBtYXBwZWQgdG8gb25seSBvbmUgdXNlciBhZGRyZXNzIHNwYWNlKS4KPj4+Pj4+ Pj4gV3JpdGVzIHRvIHRoZSBkb29yYmVsbCBhcGVydHVyZSBhcmUgaW50ZXJjZXB0ZWQgYnkgR1BV IGhhcmR3YXJlLCBhbGxvd2luZwo+Pj4+Pj4+PiB1c2Vyc3BhY2UgY29kZSB0byBzYWZlbHkgbWFu YWdlIHdvcmsgcXVldWVzIChyaW5ncykgd2l0aG91dCByZXF1aXJpbmcgYQo+Pj4+Pj4+PiBrZXJu ZWwgY2FsbCBmb3IgZXZlcnkgcmluZyB1cGRhdGUuCj4+Pj4+Pj4+IEZpcnN0IHN0ZXAgZm9yIGFu IGFwcGxpY2F0aW9uIHByb2Nlc3MgaXMgdG8gb3BlbiB0aGUga2ZkIGRldmljZS4gQ2FsbHMgdG8K Pj4+Pj4+Pj4gb3BlbiBjcmVhdGUgYSBrZmQgInByb2Nlc3MiIHN0cnVjdHVyZSBvbmx5IGZvciB0 aGUgZmlyc3QgdGhyZWFkIG9mIHRoZQo+Pj4+Pj4+PiBwcm9jZXNzLiBTdWJzZXF1ZW50IG9wZW4g Y2FsbHMgYXJlIGNoZWNrZWQgdG8gc2VlIGlmIHRoZXkgYXJlIGZyb20gcHJvY2Vzc2VzCj4+Pj4+ Pj4+IHVzaW5nIHRoZSBzYW1lIG1tX3N0cnVjdCBhbmQsIGlmIHNvLCBkb24ndCBkbyBhbnl0aGlu Zy4gVGhlIGtmZCBwZXItcHJvY2Vzcwo+Pj4+Pj4+PiBkYXRhIGxpdmVzIGFzIGxvbmcgYXMgdGhl IG1tX3N0cnVjdCBleGlzdHMuIEVhY2ggbW1fc3RydWN0IGlzIGFzc29jaWF0ZWQKPj4+Pj4+Pj4g d2l0aCBhIHVuaXF1ZSBQQVNJRCwgYWxsb3dpbmcgdGhlIElPTU1VdjIgdG8gbWFrZSB1c2Vyc3Bh Y2UgcHJvY2VzcyBtZW1vcnkKPj4+Pj4+Pj4gYWNjZXNzaWJsZSB0byB0aGUgR1BVLgo+Pj4+Pj4+ PiBOZXh0IHN0ZXAgaXMgZm9yIHRoZSBhcHBsaWNhdGlvbiB0byBjb2xsZWN0IHRvcG9sb2d5IGlu Zm9ybWF0aW9uIHZpYSBzeXNmcy4KPj4+Pj4+Pj4gVGhpcyBnaXZlcyB1c2Vyc3BhY2UgZW5vdWdo IGluZm9ybWF0aW9uIHRvIGJlIGFibGUgdG8gaWRlbnRpZnkgc3BlY2lmaWMKPj4+Pj4+Pj4gbm9k ZXMgKHByb2Nlc3NvcnMpIGluIHN1YnNlcXVlbnQgcXVldWUgbWFuYWdlbWVudCBjYWxscy4gQXBw bGljYXRpb24KPj4+Pj4+Pj4gcHJvY2Vzc2VzIGNhbiBjcmVhdGUgcXVldWVzIG9uIG11bHRpcGxl IHByb2Nlc3NvcnMsIGFuZCBwcm9jZXNzb3JzIHN1cHBvcnQKPj4+Pj4+Pj4gcXVldWVzIGZyb20g bXVsdGlwbGUgcHJvY2Vzc2VzLgo+Pj4+Pj4+PiBBdCB0aGlzIHBvaW50IHRoZSBhcHBsaWNhdGlv biBjYW4gY3JlYXRlIHdvcmsgcXVldWVzIGluIHVzZXJzcGFjZSBtZW1vcnkgYW5kCj4+Pj4+Pj4+ IHBhc3MgdGhlbSB0aHJvdWdoIHRoZSB1c2VybW9kZSBsaWJyYXJ5IHRvIGtmZCB0byBoYXZlIHRo ZW0gbWFwcGVkIG9udG8gSFcKPj4+Pj4+Pj4gcXVldWUgc2xvdHMgc28gdGhhdCBjb21tYW5kcyB3 cml0dGVuIHRvIHRoZSBxdWV1ZXMgY2FuIGJlIGV4ZWN1dGVkIGJ5IHRoZQo+Pj4+Pj4+PiBHUFUu IFF1ZXVlIG9wZXJhdGlvbnMgc3BlY2lmeSBhIHByb2Nlc3NvciBub2RlLCBhbmQgc28gdGhlIGJ1 bGsgb2YgdGhpcyBjb2RlCj4+Pj4+Pj4+IGlzIGRldmljZS1zcGVjaWZpYy4KPj4+Pj4+Pj4gV3Jp dHRlbiBieSBKb2huIEJyaWRnbWFuIDxKb2huLkJyaWRnbWFuQGFtZC5jb20+Cj4+Pj4+Pj4+Cj4+ Pj4+Pj4+Cj4+Pj4+Pj4+IEFsZXhleSBTa2lkYW5vdiAoMSk6Cj4+Pj4+Pj4+ICAgYW1ka2ZkOiBJ bXBsZW1lbnQgdGhlIEdldCBQcm9jZXNzIEFwZXJ0dXJlIElPQ1RMCj4+Pj4+Pj4+Cj4+Pj4+Pj4+ IEFuZHJldyBMZXd5Y2t5ICgzKToKPj4+Pj4+Pj4gICBhbWRrZmQ6IEFkZCBiYXNpYyBtb2R1bGVz IHRvIGFtZGtmZAo+Pj4+Pj4+PiAgIGFtZGtmZDogQWRkIGludGVycnVwdCBoYW5kbGluZyBtb2R1 bGUKPj4+Pj4+Pj4gICBhbWRrZmQ6IEltcGxlbWVudCB0aGUgU2V0IE1lbW9yeSBQb2xpY3kgSU9D VEwKPj4+Pj4+Pj4KPj4+Pj4+Pj4gQmVuIEdveiAoOCk6Cj4+Pj4+Pj4+ICAgYW1ka2ZkOiBBZGQg cXVldWUgbW9kdWxlCj4+Pj4+Pj4+ICAgYW1ka2ZkOiBBZGQgbXFkX21hbmFnZXIgbW9kdWxlCj4+ Pj4+Pj4+ICAgYW1ka2ZkOiBBZGQga2VybmVsIHF1ZXVlIG1vZHVsZQo+Pj4+Pj4+PiAgIGFtZGtm ZDogQWRkIG1vZHVsZSBwYXJhbWV0ZXIgb2Ygc2NoZWR1bGluZyBwb2xpY3kKPj4+Pj4+Pj4gICBh bWRrZmQ6IEFkZCBwYWNrZXQgbWFuYWdlciBtb2R1bGUKPj4+Pj4+Pj4gICBhbWRrZmQ6IEFkZCBw cm9jZXNzIHF1ZXVlIG1hbmFnZXIgbW9kdWxlCj4+Pj4+Pj4+ICAgYW1ka2ZkOiBBZGQgZGV2aWNl IHF1ZXVlIG1hbmFnZXIgbW9kdWxlCj4+Pj4+Pj4+ICAgYW1ka2ZkOiBJbXBsZW1lbnQgdGhlIGNy ZWF0ZS9kZXN0cm95L3VwZGF0ZSBxdWV1ZSBJT0NUTHMKPj4+Pj4+Pj4KPj4+Pj4+Pj4gRXZnZW55 IFBpbmNodWsgKDMpOgo+Pj4+Pj4+PiAgIGFtZGtmZDogQWRkIHRvcG9sb2d5IG1vZHVsZSB0byBh bWRrZmQKPj4+Pj4+Pj4gICBhbWRrZmQ6IEltcGxlbWVudCB0aGUgR2V0IENsb2NrIENvdW50ZXJz IElPQ1RMCj4+Pj4+Pj4+ICAgYW1ka2ZkOiBJbXBsZW1lbnQgdGhlIFBNQyBBY3F1aXJlL1JlbGVh c2UgSU9DVExzCj4+Pj4+Pj4+Cj4+Pj4+Pj4+IE9kZWQgR2FiYmF5ICgxMCk6Cj4+Pj4+Pj4+ICAg bW06IEFkZCBrZmRfcHJvY2VzcyBwb2ludGVyIHRvIG1tX3N0cnVjdAo+Pj4+Pj4+PiAgIGRybS9y YWRlb246IHJlZHVjZSBudW1iZXIgb2YgZnJlZSBWTUlEcyBhbmQgcGlwZXMgaW4gS1YKPj4+Pj4+ Pj4gICBkcm0vcmFkZW9uL2NpazogRG9uJ3QgdG91Y2ggaW50IG9mIHBpcGVzIDEtNwo+Pj4+Pj4+ PiAgIGRybS9yYWRlb246IFJlcG9ydCBkb29yYmVsbCBjb25maWd1cmF0aW9uIHRvIGFtZGtmZAo+ Pj4+Pj4+PiAgIGRybS9yYWRlb246IGFkZGluZyBzeW5jaHJvbml6YXRpb24gZm9yIEdSQk0gR0ZY Cj4+Pj4+Pj4+ICAgZHJtL3JhZGVvbjogQWRkIHJhZGVvbiA8LS0+IGFtZGtmZCBpbnRlcmZhY2UK Pj4+Pj4+Pj4gICBVcGRhdGUgTUFJTlRBSU5FUlMgYW5kIENSRURJVFMgZmlsZXMgd2l0aCBhbWRr ZmQgaW5mbwo+Pj4+Pj4+PiAgIGFtZGtmZDogQWRkIElPQ1RMIHNldCBkZWZpbml0aW9ucyBvZiBh bWRrZmQKPj4+Pj4+Pj4gICBhbWRrZmQ6IEFkZCBhbWRrZmQgc2tlbGV0b24gZHJpdmVyCj4+Pj4+ Pj4+ICAgYW1ka2ZkOiBBZGQgYmluZGluZy91bmJpbmRpbmcgY2FsbHMgdG8gYW1kX2lvbW11IGRy aXZlcgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiAgQ1JFRElUUyAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgfCAgICA3ICsKPj4+Pj4+Pj4gIE1BSU5UQUlORVJTICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAxMCArCj4+Pj4+Pj4+ICBkcml2ZXJz L2dwdS9kcm0vcmFkZW9uL0tjb25maWcgICAgICAgICAgICAgICAgICAgICB8ICAgIDIgKwo+Pj4+ Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9NYWtlZmlsZSAgICAgICAgICAgICAgICAgICAg fCAgICAzICsKPj4+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL0tjb25maWcg ICAgICAgICAgICAgIHwgICAxMCArCj4+Pj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2Ft ZGtmZC9NYWtlZmlsZSAgICAgICAgICAgICB8ICAgMTQgKwo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUv ZHJtL3JhZGVvbi9hbWRrZmQvY2lrX21xZHMuaCAgICAgICAgICAgfCAgMTg1ICsrKwo+Pj4+Pj4+ PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQvY2lrX3JlZ3MuaCAgICAgICAgICAgfCAg MjIwICsrKysKPj4+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF9hcGVy dHVyZS5jICAgICAgIHwgIDEyMyArKwo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9h bWRrZmQva2ZkX2NoYXJkZXYuYyAgICAgICAgfCAgNTE4ICsrKysrKysrKwo+Pj4+Pj4+PiAgZHJp dmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2NyYXQuaCAgICAgICAgICAgfCAgMjk0ICsr KysrCj4+Pj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfZGV2aWNlLmMg ICAgICAgICB8ICAyNTQgKysrKwo+Pj4+Pj4+PiAgLi4uL2RybS9yYWRlb24vYW1ka2ZkL2tmZF9k ZXZpY2VfcXVldWVfbWFuYWdlci5jICAgfCAgOTg1ICsrKysrKysrKysrKysrKysKPj4+Pj4+Pj4g IC4uLi9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfZGV2aWNlX3F1ZXVlX21hbmFnZXIuaCAgIHwgIDEw MSArKwo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2Rvb3JiZWxs LmMgICAgICAgfCAgMjY0ICsrKysrCj4+Pj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2Ft ZGtmZC9rZmRfaW50ZXJydXB0LmMgICAgICB8ICAxNjEgKysrCj4+Pj4+Pj4+ICBkcml2ZXJzL2dw dS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfa2VybmVsX3F1ZXVlLmMgICB8ICAzMDUgKysrKysKPj4+ Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF9rZXJuZWxfcXVldWUuaCAg IHwgICA2NiArKwo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX21v ZHVsZS5jICAgICAgICAgfCAgMTMxICsrKwo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVv bi9hbWRrZmQva2ZkX21xZF9tYW5hZ2VyLmMgICAgfCAgMjkxICsrKysrCj4+Pj4+Pj4+ICBkcml2 ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfbXFkX21hbmFnZXIuaCAgICB8ICAgNTQgKwo+ Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3BhY2tldF9tYW5hZ2Vy LmMgfCAgNDg4ICsrKysrKysrCj4+Pj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtm ZC9rZmRfcGFzaWQuYyAgICAgICAgICB8ICAgOTcgKysKPj4+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2Ry bS9yYWRlb24vYW1ka2ZkL2tmZF9wbTRfaGVhZGVycy5oICAgIHwgIDY4MiArKysrKysrKysrKwo+ Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3BtNF9vcGNvZGVzLmgg ICAgfCAgMTA3ICsrCj4+Pj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRf cHJpdi5oICAgICAgICAgICB8ICA0NjYgKysrKysrKysKPj4+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2Ry bS9yYWRlb24vYW1ka2ZkL2tmZF9wcm9jZXNzLmMgICAgICAgIHwgIDQwNSArKysrKysrCj4+Pj4+ Pj4+ICAuLi4vZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3Byb2Nlc3NfcXVldWVfbWFuYWdlci5jICB8 ICAzNDMgKysrKysrCj4+Pj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRf cXVldWUuYyAgICAgICAgICB8ICAxMDkgKysKPj4+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRl b24vYW1ka2ZkL2tmZF90b3BvbG9neS5jICAgICAgIHwgMTIwNwo+Pj4+Pj4+PiArKysrKysrKysr KysrKysrKysrKwo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3Rv cG9sb2d5LmggICAgICAgfCAgMTY4ICsrKwo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVv bi9hbWRrZmQva2ZkX3ZpZG1lbS5jICAgICAgICAgfCAgIDk2ICsrCj4+Pj4+Pj4+ICBkcml2ZXJz L2dwdS9kcm0vcmFkZW9uL2Npay5jICAgICAgICAgICAgICAgICAgICAgICB8ICAxNTQgKy0tCj4+ Pj4+Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2Npa19yZWcuaCAgICAgICAgICAgICAgICAg ICB8ICAgNjUgKysKPj4+Pj4+Pj4gIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vY2lrZC5oICAgICAg ICAgICAgICAgICAgICAgIHwgICA1MSArLQo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVv bi9yYWRlb24uaCAgICAgICAgICAgICAgICAgICAgfCAgICA5ICsKPj4+Pj4+Pj4gIGRyaXZlcnMv Z3B1L2RybS9yYWRlb24vcmFkZW9uX2RldmljZS5jICAgICAgICAgICAgIHwgICAzMiArCj4+Pj4+ Pj4+ICBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL3JhZGVvbl9kcnYuYyAgICAgICAgICAgICAgICB8 ICAgIDUgKwo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9yYWRlb25fa2ZkLmMgICAg ICAgICAgICAgICAgfCAgNTY2ICsrKysrKysrKwo+Pj4+Pj4+PiAgZHJpdmVycy9ncHUvZHJtL3Jh ZGVvbi9yYWRlb25fa2ZkLmggICAgICAgICAgICAgICAgfCAgMTE5ICsrCj4+Pj4+Pj4+ICBkcml2 ZXJzL2dwdS9kcm0vcmFkZW9uL3JhZGVvbl9rbXMuYyAgICAgICAgICAgICAgICB8ICAgIDcgKwo+ Pj4+Pj4+PiAgaW5jbHVkZS9saW51eC9tbV90eXBlcy5oICAgICAgICAgICAgICAgICAgICAgICAg ICAgfCAgIDE0ICsKPj4+Pj4+Pj4gIGluY2x1ZGUvdWFwaS9saW51eC9rZmRfaW9jdGwuaCAgICAg ICAgICAgICAgICAgICAgIHwgIDEzMyArKysKPj4+Pj4+Pj4gIDQzIGZpbGVzIGNoYW5nZWQsIDky MjYgaW5zZXJ0aW9ucygrKSwgOTUgZGVsZXRpb25zKC0pCj4+Pj4+Pj4+ICBjcmVhdGUgbW9kZSAx MDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQvS2NvbmZpZwo+Pj4+Pj4+PiAgY3Jl YXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL01ha2VmaWxlCj4+ Pj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQv Y2lrX21xZHMuaAo+Pj4+Pj4+PiAgY3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9y YWRlb24vYW1ka2ZkL2Npa19yZWdzLmgKPj4+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2 ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfYXBlcnR1cmUuYwo+Pj4+Pj4+PiAgY3JlYXRl IG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF9jaGFyZGV2LmMK Pj4+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtm ZC9rZmRfY3JhdC5oCj4+Pj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJt L3JhZGVvbi9hbWRrZmQva2ZkX2RldmljZS5jCj4+Pj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQg ZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2RldmljZV9xdWV1ZV9tYW5hZ2VyLmMK Pj4+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtm ZC9rZmRfZGV2aWNlX3F1ZXVlX21hbmFnZXIuaAo+Pj4+Pj4+PiAgY3JlYXRlIG1vZGUgMTAwNjQ0 IGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF9kb29yYmVsbC5jCj4+Pj4+Pj4+ICBj cmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2ludGVy cnVwdC5jCj4+Pj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVv bi9hbWRrZmQva2ZkX2tlcm5lbF9xdWV1ZS5jCj4+Pj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQg ZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX2tlcm5lbF9xdWV1ZS5oCj4+Pj4+Pj4+ ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX21v ZHVsZS5jCj4+Pj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVv bi9hbWRrZmQva2ZkX21xZF9tYW5hZ2VyLmMKPj4+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBk cml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfbXFkX21hbmFnZXIuaAo+Pj4+Pj4+PiAg Y3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF9wYWNr ZXRfbWFuYWdlci5jCj4+Pj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJt L3JhZGVvbi9hbWRrZmQva2ZkX3Bhc2lkLmMKPj4+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBk cml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfcG00X2hlYWRlcnMuaAo+Pj4+Pj4+PiAg Y3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF9wbTRf b3Bjb2Rlcy5oCj4+Pj4+Pj4+ICBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3Jh ZGVvbi9hbWRrZmQva2ZkX3ByaXYuaAo+Pj4+Pj4+PiAgY3JlYXRlIG1vZGUgMTAwNjQ0IGRyaXZl cnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF9wcm9jZXNzLmMKPj4+Pj4+Pj4gIGNyZWF0ZSBt b2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2FtZGtmZC9rZmRfcHJvY2Vzc19xdWV1 ZV9tYW5hZ2VyLmMKPj4+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0v cmFkZW9uL2FtZGtmZC9rZmRfcXVldWUuYwo+Pj4+Pj4+PiAgY3JlYXRlIG1vZGUgMTAwNjQ0IGRy aXZlcnMvZ3B1L2RybS9yYWRlb24vYW1ka2ZkL2tmZF90b3BvbG9neS5jCj4+Pj4+Pj4+ICBjcmVh dGUgbW9kZSAxMDA2NDQgZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9hbWRrZmQva2ZkX3RvcG9sb2d5 LmgKPj4+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dwdS9kcm0vcmFkZW9uL2Ft ZGtmZC9rZmRfdmlkbWVtLmMKPj4+Pj4+Pj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2dw dS9kcm0vcmFkZW9uL3JhZGVvbl9rZmQuYwo+Pj4+Pj4+PiAgY3JlYXRlIG1vZGUgMTAwNjQ0IGRy aXZlcnMvZ3B1L2RybS9yYWRlb24vcmFkZW9uX2tmZC5oCj4+Pj4+Pj4+ICBjcmVhdGUgbW9kZSAx MDA2NDQgaW5jbHVkZS91YXBpL2xpbnV4L2tmZF9pb2N0bC5oCj4+Pj4+Pj4+Cj4+Pj4+Pj4+IC0t Cj4+Pj4+Pj4+IDEuOS4xCj4+Pj4+Pj4+Cj4+Pj4+Pgo+Pj4+Pgo+Pj4+Cj4+CgpfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBs aXN0CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cDovL2xpc3RzLmZyZWVkZXNr dG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RyaS1kZXZlbAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qa0-f43.google.com (mail-qa0-f43.google.com [209.85.216.43]) by kanga.kvack.org (Postfix) with ESMTP id 71E596B006C for ; Mon, 21 Jul 2014 14:36:57 -0400 (EDT) Received: by mail-qa0-f43.google.com with SMTP id w8so5580082qac.30 for ; Mon, 21 Jul 2014 11:36:57 -0700 (PDT) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0187.outbound.protection.outlook.com. [207.46.163.187]) by mx.google.com with ESMTPS id m15si30221543qav.131.2014.07.21.11.36.56 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 21 Jul 2014 11:36:56 -0700 (PDT) Message-ID: <53CD5DBC.7010301@amd.com> Date: Mon, 21 Jul 2014 21:36:44 +0300 From: Oded Gabbay MIME-Version: 1.0 Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver References: <53C7D645.3070607@amd.com> <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <53CD1FB6.1000602@amd.com> <20140721155437.GA4519@gmail.com> <53CD5122.5040804@amd.com> <20140721181433.GA5196@gmail.com> In-Reply-To: <20140721181433.GA5196@gmail.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Jerome Glisse Cc: =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , David Airlie , Alex Deucher , Andrew Morton , John Bridgman , Joerg Roedel , Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , Ben Goz , Alexey Skidanov , Evgeny Pinchuk , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm On 21/07/14 21:14, Jerome Glisse wrote: > On Mon, Jul 21, 2014 at 08:42:58PM +0300, Oded Gabbay wrote: >> On 21/07/14 18:54, Jerome Glisse wrote: >>> On Mon, Jul 21, 2014 at 05:12:06PM +0300, Oded Gabbay wrote: >>>> On 21/07/14 16:39, Christian K=C3=B6nig wrote: >>>>> Am 21.07.2014 14:36, schrieb Oded Gabbay: >>>>>> On 20/07/14 20:46, Jerome Glisse wrote: >>>>>>> On Thu, Jul 17, 2014 at 04:57:25PM +0300, Oded Gabbay wrote: >>>>>>>> Forgot to cc mailing list on cover letter. Sorry. >>>>>>>> >>>>>>>> As a continuation to the existing discussion, here is a v2 patch= series >>>>>>>> restructured with a cleaner history and no totally-different-ear= ly-versions >>>>>>>> of the code. >>>>>>>> >>>>>>>> Instead of 83 patches, there are now a total of 25 patches, wher= e 5 of them >>>>>>>> are modifications to radeon driver and 18 of them include only a= mdkfd code. >>>>>>>> There is no code going away or even modified between patches, on= ly added. >>>>>>>> >>>>>>>> The driver was renamed from radeon_kfd to amdkfd and moved to re= side under >>>>>>>> drm/radeon/amdkfd. This move was done to emphasize the fact that= this driver >>>>>>>> is an AMD-only driver at this point. Having said that, we do for= esee a >>>>>>>> generic hsa framework being implemented in the future and in tha= t case, we >>>>>>>> will adjust amdkfd to work within that framework. >>>>>>>> >>>>>>>> As the amdkfd driver should support multiple AMD gfx drivers, we= want to >>>>>>>> keep it as a seperate driver from radeon. Therefore, the amdkfd = code is >>>>>>>> contained in its own folder. The amdkfd folder was put under the= radeon >>>>>>>> folder because the only AMD gfx driver in the Linux kernel at th= is point >>>>>>>> is the radeon driver. Having said that, we will probably need to= move it >>>>>>>> (maybe to be directly under drm) after we integrate with additio= nal AMD gfx >>>>>>>> drivers. >>>>>>>> >>>>>>>> For people who like to review using git, the v2 patch set is loc= ated at: >>>>>>>> http://cgit.freedesktop.org/~gabbayo/linux/log/?h=3Dkfd-next-3.1= 7-v2 >>>>>>>> >>>>>>>> Written by Oded Gabbayh >>>>>>> >>>>>>> So quick comments before i finish going over all patches. There i= s many >>>>>>> things that need more documentation espacialy as of right now the= re is >>>>>>> no userspace i can go look at. >>>>>> So quick comments on some of your questions but first of all, than= ks for the >>>>>> time you dedicated to review the code. >>>>>>> >>>>>>> There few show stopper, biggest one is gpu memory pinning this is= a big >>>>>>> no, that would need serious arguments for any hope of convincing = me on >>>>>>> that side. >>>>>> We only do gpu memory pinning for kernel objects. There are no use= rspace >>>>>> objects that are pinned on the gpu memory in our driver. If that i= s the case, >>>>>> is it still a show stopper ? >>>>>> >>>>>> The kernel objects are: >>>>>> - pipelines (4 per device) >>>>>> - mqd per hiq (only 1 per device) >>>>>> - mqd per userspace queue. On KV, we support up to 1K queues per p= rocess, for >>>>>> a total of 512K queues. Each mqd is 151 bytes, but the allocation = is done in >>>>>> 256 alignment. So total *possible* memory is 128MB >>>>>> - kernel queue (only 1 per device) >>>>>> - fence address for kernel queue >>>>>> - runlists for the CP (1 or 2 per device) >>>>> >>>>> The main questions here are if it's avoid able to pin down the memo= ry and if the >>>>> memory is pinned down at driver load, by request from userspace or = by anything >>>>> else. >>>>> >>>>> As far as I can see only the "mqd per userspace queue" might be a b= it >>>>> questionable, everything else sounds reasonable. >>>>> >>>>> Christian. >>>> >>>> Most of the pin downs are done on device initialization. >>>> The "mqd per userspace" is done per userspace queue creation. Howeve= r, as I >>>> said, it has an upper limit of 128MB on KV, and considering the 2G l= ocal >>>> memory, I think it is OK. >>>> The runlists are also done on userspace queue creation/deletion, but= we only >>>> have 1 or 2 runlists per device, so it is not that bad. >>> >>> 2G local memory ? You can not assume anything on userside configurati= on some >>> one might build an hsa computer with 512M and still expect a function= ing >>> desktop. >> First of all, I'm only considering Kaveri computer, not "hsa" computer= . >> Second, I would imagine we can build some protection around it, like >> checking total local memory and limit number of queues based on some >> percentage of that total local memory. So, if someone will have only >> 512M, he will be able to open less queues. >> >> >>> >>> I need to go look into what all this mqd is for, what it does and wha= t it is >>> about. But pinning is really bad and this is an issue with userspace = command >>> scheduling an issue that obviously AMD fails to take into account in = design >>> phase. >> Maybe, but that is the H/W design non-the-less. We can't very well >> change the H/W. >=20 > You can not change the hardware but it is not an excuse to allow bad de= sign to > sneak in software to work around that. So i would rather penalize bad h= ardware > design and have command submission in the kernel, until AMD fix its har= dware to > allow proper scheduling by the kernel and proper control by the kernel.= =20 I'm sorry but I do *not* think this is a bad design. S/W scheduling in the kernel can not, IMO, scale well to 100K queues and 10K processes. > Because really where we want to go is having GPU closer to a CPU in ter= m of scheduling > capacity and once we get there we want the kernel to always be able to = take over > and do whatever it wants behind process back. Who do you refer to when you say "we" ? AFAIK, the hw scheduling direction is where AMD is now and where it is heading in the future. That doesn't preclude the option to allow the kernel to take over and do what he wants. I agree that in KV we have a problem where we can't do a mid-wave preemption, so theoretically, a long running compute kernel can make things messy, but in Carrizo, we will have this ability. Having said that, it will only be through the CP H/W scheduling. So AMD is _not_ going to abandon H/W scheduling. You can dislike it, but this is the situation. >=20 >>>>> >>>>>>> >>>>>>> It might be better to add a drivers/gpu/drm/amd directory and add= common >>>>>>> stuff there. >>>>>>> >>>>>>> Given that this is not intended to be final HSA api AFAICT then i= would >>>>>>> say this far better to avoid the whole kfd module and add ioctl t= o radeon. >>>>>>> This would avoid crazy communication btw radeon and kfd. >>>>>>> >>>>>>> The whole aperture business needs some serious explanation. Espec= ialy as >>>>>>> you want to use userspace address there is nothing to prevent use= rspace >>>>>>> program from allocating things at address you reserve for lds, sc= ratch, >>>>>>> ... only sane way would be to move those lds, scratch inside the = virtual >>>>>>> address reserved for kernel (see kernel memory map). >>>>>>> >>>>>>> The whole business of locking performance counter for exclusive p= er process >>>>>>> access is a big NO. Which leads me to the questionable usefullnes= s of user >>>>>>> space command ring. >>>>>> That's like saying: "Which leads me to the questionable usefulness= of HSA". I >>>>>> find it analogous to a situation where a network maintainer nackin= g a driver >>>>>> for a network card, which is slower than a different network card.= Doesn't >>>>>> seem reasonable this situation is would happen. He would still put= both the >>>>>> drivers in the kernel because people want to use the H/W and its f= eatures. So, >>>>>> I don't think this is a valid reason to NACK the driver. >>> >>> Let me rephrase, drop the the performance counter ioctl and modulo me= mory pinning >>> i see no objection. In other word, i am not NACKING whole patchset i = am NACKING >>> the performance ioctl. >>> >>> Again this is another argument for round trip to the kernel. As insid= e kernel you >>> could properly do exclusive gpu counter access accross single user cm= d buffer >>> execution. >>> >>>>>> >>>>>>> I only see issues with that. First and foremost i would >>>>>>> need to see solid figures that kernel ioctl or syscall has a high= er an >>>>>>> overhead that is measurable in any meaning full way against a sim= ple >>>>>>> function call. I know the userspace command ring is a big marketi= ng features >>>>>>> that please ignorant userspace programmer. But really this only b= rings issues >>>>>>> and for absolutely not upside afaict. >>>>>> Really ? You think that doing a context switch to kernel space, wi= th all its >>>>>> overhead, is _not_ more expansive than just calling a function in = userspace >>>>>> which only puts a buffer on a ring and writes a doorbell ? >>> >>> I am saying the overhead is not that big and it probably will not mat= ter in most >>> usecase. For instance i did wrote the most useless kernel module that= add two >>> number through an ioctl (http://people.freedesktop.org/~glisse/adder.= tar) and >>> it takes ~0.35microseconds with ioctl while function is ~0.025microse= conds so >>> ioctl is 13 times slower. >>> >>> Now if there is enough data that shows that a significant percentage = of jobs >>> submited to the GPU will take less that 0.35microsecond then yes user= space >>> scheduling does make sense. But so far all we have is handwaving with= no data >>> to support any facts. >>> >>> >>> Now if we want to schedule from userspace than you will need to do so= mething >>> about the pinning, something that gives control to kernel so that ker= nel can >>> unpin when it wants and move object when it wants no matter what user= space is >>> doing. >>> >>>>>>> >>>>>>> So i would rather see a very simple ioctl that write the doorbell= and might >>>>>>> do more than that in case of ring/queue overcommit where it would= first have >>>>>>> to wait for a free ring/queue to schedule stuff. This would also = allow sane >>>>>>> implementation of things like performance counter that could be a= cquire by >>>>>>> kernel for duration of a job submitted by userspace. While still = not optimal >>>>>>> this would be better that userspace locking. >>>>>>> >>>>>>> >>>>>>> I might have more thoughts once i am done with all the patches. >>>>>>> >>>>>>> Cheers, >>>>>>> J=C3=A9r=C3=B4me >>>>>>> >>>>>>>> >>>>>>>> Original Cover Letter: >>>>>>>> >>>>>>>> This patch set implements a Heterogeneous System Architecture (H= SA) driver >>>>>>>> for radeon-family GPUs. >>>>>>>> HSA allows different processor types (CPUs, DSPs, GPUs, etc..) t= o share >>>>>>>> system resources more effectively via HW features including shar= ed pageable >>>>>>>> memory, userspace-accessible work queues, and platform-level ato= mics. In >>>>>>>> addition to the memory protection mechanisms in GPUVM and IOMMUv= 2, the Sea >>>>>>>> Islands family of GPUs also performs HW-level validation of comm= ands passed >>>>>>>> in through the queues (aka rings). >>>>>>>> >>>>>>>> The code in this patch set is intended to serve both as a sample= driver for >>>>>>>> other HSA-compatible hardware devices and as a production driver= for >>>>>>>> radeon-family processors. The code is architected to support mul= tiple CPUs >>>>>>>> each with connected GPUs, although the current implementation fo= cuses on a >>>>>>>> single Kaveri/Berlin APU, and works alongside the existing radeo= n kernel >>>>>>>> graphics driver (kgd). >>>>>>>> AMD GPUs designed for use with HSA (Sea Islands and up) share so= me hardware >>>>>>>> functionality between HSA compute and regular gfx/compute (memor= y, >>>>>>>> interrupts, registers), while other functionality has been added >>>>>>>> specifically for HSA compute (hw scheduler for virtualized comp= ute rings). >>>>>>>> All shared hardware is owned by the radeon graphics driver, and = an interface >>>>>>>> between kfd and kgd allows the kfd to make use of those shared r= esources, >>>>>>>> while HSA-specific functionality is managed directly by kfd by s= ubmitting >>>>>>>> packets into an HSA-specific command queue (the "HIQ"). >>>>>>>> >>>>>>>> During kfd module initialization a char device node (/dev/kfd) i= s created >>>>>>>> (surviving until module exit), with ioctls for queue creation & = management, >>>>>>>> and data structures are initialized for managing HSA device topo= logy. >>>>>>>> The rest of the initialization is driven by calls from the radeo= n kgd at the >>>>>>>> following points : >>>>>>>> >>>>>>>> - radeon_init (kfd_init) >>>>>>>> - radeon_exit (kfd_fini) >>>>>>>> - radeon_driver_load_kms (kfd_device_probe, kfd_device_init) >>>>>>>> - radeon_driver_unload_kms (kfd_device_fini) >>>>>>>> >>>>>>>> During the probe and init processing per-device data structures = are >>>>>>>> established which connect to the associated graphics kernel driv= er. This >>>>>>>> information is exposed to userspace via sysfs, along with a vers= ion number >>>>>>>> allowing userspace to determine if a topology change has occurre= d while it >>>>>>>> was reading from sysfs. >>>>>>>> The interface between kfd and kgd also allows the kfd to request= buffer >>>>>>>> management services from kgd, and allows kgd to route interrupt = requests to >>>>>>>> kfd code since the interrupt block is shared between regular >>>>>>>> graphics/compute and HSA compute subsystems in the GPU. >>>>>>>> >>>>>>>> The kfd code works with an open source usermode library ("libhsa= kmt") which >>>>>>>> is in the final stages of IP review and should be published in a= separate >>>>>>>> repo over the next few days. >>>>>>>> The code operates in one of three modes, selectable via the sche= d_policy >>>>>>>> module parameter : >>>>>>>> >>>>>>>> - sched_policy=3D0 uses a hardware scheduler running in the MEC = block within >>>>>>>> CP, and allows oversubscription (more queues than HW slots) >>>>>>>> - sched_policy=3D1 also uses HW scheduling but does not allow >>>>>>>> oversubscription, so create_queue requests fail when we run out = of HW slots >>>>>>>> - sched_policy=3D2 does not use HW scheduling, so the driver man= ually assigns >>>>>>>> queues to HW slots by programming registers >>>>>>>> >>>>>>>> The "no HW scheduling" option is for debug & new hardware bringu= p only, so >>>>>>>> has less test coverage than the other options. Default in the cu= rrent code >>>>>>>> is "HW scheduling without oversubscription" since that is where = we have the >>>>>>>> most test coverage but we expect to change the default to "HW sc= heduling >>>>>>>> with oversubscription" after further testing. This effectively r= emoves the >>>>>>>> HW limit on the number of work queues available to applications. >>>>>>>> >>>>>>>> Programs running on the GPU are associated with an address space= through the >>>>>>>> VMID field, which is translated to a unique PASID at access time= via a set >>>>>>>> of 16 VMID-to-PASID mapping registers. The available VMIDs (curr= ently 16) >>>>>>>> are partitioned (under control of the radeon kgd) between curren= t >>>>>>>> gfx/compute and HSA compute, with each getting 8 in the current = code. The >>>>>>>> VMID-to-PASID mapping registers are updated by the HW scheduler = when used, >>>>>>>> and by driver code if HW scheduling is not being used. >>>>>>>> The Sea Islands compute queues use a new "doorbell" mechanism in= stead of the >>>>>>>> earlier kernel-managed write pointer registers. Doorbells use a = separate BAR >>>>>>>> dedicated for this purpose, and pages within the doorbell apertu= re are >>>>>>>> mapped to userspace (each page mapped to only one user address s= pace). >>>>>>>> Writes to the doorbell aperture are intercepted by GPU hardware,= allowing >>>>>>>> userspace code to safely manage work queues (rings) without requ= iring a >>>>>>>> kernel call for every ring update. >>>>>>>> First step for an application process is to open the kfd device.= Calls to >>>>>>>> open create a kfd "process" structure only for the first thread = of the >>>>>>>> process. Subsequent open calls are checked to see if they are fr= om processes >>>>>>>> using the same mm_struct and, if so, don't do anything. The kfd = per-process >>>>>>>> data lives as long as the mm_struct exists. Each mm_struct is as= sociated >>>>>>>> with a unique PASID, allowing the IOMMUv2 to make userspace proc= ess memory >>>>>>>> accessible to the GPU. >>>>>>>> Next step is for the application to collect topology information= via sysfs. >>>>>>>> This gives userspace enough information to be able to identify s= pecific >>>>>>>> nodes (processors) in subsequent queue management calls. Applica= tion >>>>>>>> processes can create queues on multiple processors, and processo= rs support >>>>>>>> queues from multiple processes. >>>>>>>> At this point the application can create work queues in userspac= e memory and >>>>>>>> pass them through the usermode library to kfd to have them mappe= d onto HW >>>>>>>> queue slots so that commands written to the queues can be execut= ed by the >>>>>>>> GPU. Queue operations specify a processor node, and so the bulk = of this code >>>>>>>> is device-specific. >>>>>>>> Written by John Bridgman >>>>>>>> >>>>>>>> >>>>>>>> Alexey Skidanov (1): >>>>>>>> amdkfd: Implement the Get Process Aperture IOCTL >>>>>>>> >>>>>>>> Andrew Lewycky (3): >>>>>>>> amdkfd: Add basic modules to amdkfd >>>>>>>> amdkfd: Add interrupt handling module >>>>>>>> amdkfd: Implement the Set Memory Policy IOCTL >>>>>>>> >>>>>>>> Ben Goz (8): >>>>>>>> amdkfd: Add queue module >>>>>>>> amdkfd: Add mqd_manager module >>>>>>>> amdkfd: Add kernel queue module >>>>>>>> amdkfd: Add module parameter of scheduling policy >>>>>>>> amdkfd: Add packet manager module >>>>>>>> amdkfd: Add process queue manager module >>>>>>>> amdkfd: Add device queue manager module >>>>>>>> amdkfd: Implement the create/destroy/update queue IOCTLs >>>>>>>> >>>>>>>> Evgeny Pinchuk (3): >>>>>>>> amdkfd: Add topology module to amdkfd >>>>>>>> amdkfd: Implement the Get Clock Counters IOCTL >>>>>>>> amdkfd: Implement the PMC Acquire/Release IOCTLs >>>>>>>> >>>>>>>> Oded Gabbay (10): >>>>>>>> mm: Add kfd_process pointer to mm_struct >>>>>>>> drm/radeon: reduce number of free VMIDs and pipes in KV >>>>>>>> drm/radeon/cik: Don't touch int of pipes 1-7 >>>>>>>> drm/radeon: Report doorbell configuration to amdkfd >>>>>>>> drm/radeon: adding synchronization for GRBM GFX >>>>>>>> drm/radeon: Add radeon <--> amdkfd interface >>>>>>>> Update MAINTAINERS and CREDITS files with amdkfd info >>>>>>>> amdkfd: Add IOCTL set definitions of amdkfd >>>>>>>> amdkfd: Add amdkfd skeleton driver >>>>>>>> amdkfd: Add binding/unbinding calls to amd_iommu driver >>>>>>>> >>>>>>>> CREDITS | 7 + >>>>>>>> MAINTAINERS | 10 + >>>>>>>> drivers/gpu/drm/radeon/Kconfig | 2 + >>>>>>>> drivers/gpu/drm/radeon/Makefile | 3 + >>>>>>>> drivers/gpu/drm/radeon/amdkfd/Kconfig | 10 + >>>>>>>> drivers/gpu/drm/radeon/amdkfd/Makefile | 14 + >>>>>>>> drivers/gpu/drm/radeon/amdkfd/cik_mqds.h | 185 +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/cik_regs.h | 220 ++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_aperture.c | 123 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_chardev.c | 518 +++++= ++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_crat.h | 294 +++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_device.c | 254 ++++ >>>>>>>> .../drm/radeon/amdkfd/kfd_device_queue_manager.c | 985 +++++= +++++++++++ >>>>>>>> .../drm/radeon/amdkfd/kfd_device_queue_manager.h | 101 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_doorbell.c | 264 +++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_interrupt.c | 161 +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.c | 305 +++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.h | 66 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_module.c | 131 +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.c | 291 +++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.h | 54 + >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_packet_manager.c | 488 +++++= +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pasid.c | 97 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pm4_headers.h | 682 +++++= ++++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pm4_opcodes.h | 107 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_priv.h | 466 +++++= +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_process.c | 405 +++++= ++ >>>>>>>> .../drm/radeon/amdkfd/kfd_process_queue_manager.c | 343 +++++= + >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_queue.c | 109 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_topology.c | 1207 >>>>>>>> ++++++++++++++++++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_topology.h | 168 +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_vidmem.c | 96 ++ >>>>>>>> drivers/gpu/drm/radeon/cik.c | 154 +-- >>>>>>>> drivers/gpu/drm/radeon/cik_reg.h | 65 ++ >>>>>>>> drivers/gpu/drm/radeon/cikd.h | 51 +- >>>>>>>> drivers/gpu/drm/radeon/radeon.h | 9 + >>>>>>>> drivers/gpu/drm/radeon/radeon_device.c | 32 + >>>>>>>> drivers/gpu/drm/radeon/radeon_drv.c | 5 + >>>>>>>> drivers/gpu/drm/radeon/radeon_kfd.c | 566 +++++= ++++ >>>>>>>> drivers/gpu/drm/radeon/radeon_kfd.h | 119 ++ >>>>>>>> drivers/gpu/drm/radeon/radeon_kms.c | 7 + >>>>>>>> include/linux/mm_types.h | 14 + >>>>>>>> include/uapi/linux/kfd_ioctl.h | 133 +++ >>>>>>>> 43 files changed, 9226 insertions(+), 95 deletions(-) >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/Kconfig >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/Makefile >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/cik_mqds.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/cik_regs.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_aperture.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_chardev.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_crat.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device_que= ue_manager.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device_que= ue_manager.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_doorbell.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_interrupt.= c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_kernel_que= ue.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_kernel_que= ue.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_module.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manage= r.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manage= r.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_packet_man= ager.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pasid.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pm4_header= s.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pm4_opcode= s.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_priv.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_process.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_process_qu= eue_manager.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_queue.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_topology.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_topology.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_vidmem.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/radeon_kfd.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/radeon_kfd.h >>>>>>>> create mode 100644 include/uapi/linux/kfd_ioctl.h >>>>>>>> >>>>>>>> -- >>>>>>>> 1.9.1 >>>>>>>> >>>>>> >>>>> >>>> >> -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933543AbaGUSg7 (ORCPT ); Mon, 21 Jul 2014 14:36:59 -0400 Received: from mail-bn1blp0190.outbound.protection.outlook.com ([207.46.163.190]:43474 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S933404AbaGUSg5 convert rfc822-to-8bit (ORCPT ); Mon, 21 Jul 2014 14:36:57 -0400 X-WSS-ID: 0N92RPE-07-BQK-02 X-M-MSG: Message-ID: <53CD5DBC.7010301@amd.com> Date: Mon, 21 Jul 2014 21:36:44 +0300 From: Oded Gabbay Organization: AMD User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Jerome Glisse CC: =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "David Airlie" , Alex Deucher , "Andrew Morton" , John Bridgman , Joerg Roedel , Andrew Lewycky , =?UTF-8?B?TWljaGVsIETDpG56ZXI=?= , Ben Goz , Alexey Skidanov , Evgeny Pinchuk , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , linux-mm Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver References: <53C7D645.3070607@amd.com> <20140720174652.GE3068@gmail.com> <53CD0961.4070505@amd.com> <53CD17FD.3000908@vodafone.de> <53CD1FB6.1000602@amd.com> <20140721155437.GA4519@gmail.com> <53CD5122.5040804@amd.com> <20140721181433.GA5196@gmail.com> In-Reply-To: <20140721181433.GA5196@gmail.com> Content-Type: text/plain; charset="UTF-8" X-Originating-IP: [10.224.155.153] Content-Transfer-Encoding: 8BIT X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:165.204.84.221;CTRY:US;IPV:NLI;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(6009001)(428002)(51914003)(24454002)(51704005)(479174003)(189002)(199002)(80022001)(81342001)(81542001)(83322001)(19580405001)(19580395003)(65816999)(64126003)(105586002)(85306003)(77982001)(15975445006)(83506001)(4396001)(44976005)(20776003)(64706001)(65956001)(68736004)(65806001)(102836001)(47776003)(79102001)(101416001)(87936001)(15202345003)(85852003)(83072002)(110136001)(21056001)(23676002)(97736001)(31966008)(107046002)(86362001)(84676001)(87266999)(76482001)(74502001)(74662001)(33656002)(106466001)(95666004)(50466002)(36756003)(93886003)(92566001)(92726001)(99396002)(46102001)(50986999)(54356999)(76176999);DIR:OUT;SFP:;SCL:1;SRVR:BLUPR02MB034;H:atltwp01.amd.com;FPR:;MLV:sfv;PTR:InfoDomainNonexistent;MX:1;LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 0279B3DD0D Authentication-Results: spf=none (sender IP is 165.204.84.221) smtp.mailfrom=Oded.Gabbay@amd.com; X-OriginatorOrg: amd4.onmicrosoft.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 21/07/14 21:14, Jerome Glisse wrote: > On Mon, Jul 21, 2014 at 08:42:58PM +0300, Oded Gabbay wrote: >> On 21/07/14 18:54, Jerome Glisse wrote: >>> On Mon, Jul 21, 2014 at 05:12:06PM +0300, Oded Gabbay wrote: >>>> On 21/07/14 16:39, Christian König wrote: >>>>> Am 21.07.2014 14:36, schrieb Oded Gabbay: >>>>>> On 20/07/14 20:46, Jerome Glisse wrote: >>>>>>> On Thu, Jul 17, 2014 at 04:57:25PM +0300, Oded Gabbay wrote: >>>>>>>> Forgot to cc mailing list on cover letter. Sorry. >>>>>>>> >>>>>>>> As a continuation to the existing discussion, here is a v2 patch series >>>>>>>> restructured with a cleaner history and no totally-different-early-versions >>>>>>>> of the code. >>>>>>>> >>>>>>>> Instead of 83 patches, there are now a total of 25 patches, where 5 of them >>>>>>>> are modifications to radeon driver and 18 of them include only amdkfd code. >>>>>>>> There is no code going away or even modified between patches, only added. >>>>>>>> >>>>>>>> The driver was renamed from radeon_kfd to amdkfd and moved to reside under >>>>>>>> drm/radeon/amdkfd. This move was done to emphasize the fact that this driver >>>>>>>> is an AMD-only driver at this point. Having said that, we do foresee a >>>>>>>> generic hsa framework being implemented in the future and in that case, we >>>>>>>> will adjust amdkfd to work within that framework. >>>>>>>> >>>>>>>> As the amdkfd driver should support multiple AMD gfx drivers, we want to >>>>>>>> keep it as a seperate driver from radeon. Therefore, the amdkfd code is >>>>>>>> contained in its own folder. The amdkfd folder was put under the radeon >>>>>>>> folder because the only AMD gfx driver in the Linux kernel at this point >>>>>>>> is the radeon driver. Having said that, we will probably need to move it >>>>>>>> (maybe to be directly under drm) after we integrate with additional AMD gfx >>>>>>>> drivers. >>>>>>>> >>>>>>>> For people who like to review using git, the v2 patch set is located at: >>>>>>>> http://cgit.freedesktop.org/~gabbayo/linux/log/?h=kfd-next-3.17-v2 >>>>>>>> >>>>>>>> Written by Oded Gabbayh >>>>>>> >>>>>>> So quick comments before i finish going over all patches. There is many >>>>>>> things that need more documentation espacialy as of right now there is >>>>>>> no userspace i can go look at. >>>>>> So quick comments on some of your questions but first of all, thanks for the >>>>>> time you dedicated to review the code. >>>>>>> >>>>>>> There few show stopper, biggest one is gpu memory pinning this is a big >>>>>>> no, that would need serious arguments for any hope of convincing me on >>>>>>> that side. >>>>>> We only do gpu memory pinning for kernel objects. There are no userspace >>>>>> objects that are pinned on the gpu memory in our driver. If that is the case, >>>>>> is it still a show stopper ? >>>>>> >>>>>> The kernel objects are: >>>>>> - pipelines (4 per device) >>>>>> - mqd per hiq (only 1 per device) >>>>>> - mqd per userspace queue. On KV, we support up to 1K queues per process, for >>>>>> a total of 512K queues. Each mqd is 151 bytes, but the allocation is done in >>>>>> 256 alignment. So total *possible* memory is 128MB >>>>>> - kernel queue (only 1 per device) >>>>>> - fence address for kernel queue >>>>>> - runlists for the CP (1 or 2 per device) >>>>> >>>>> The main questions here are if it's avoid able to pin down the memory and if the >>>>> memory is pinned down at driver load, by request from userspace or by anything >>>>> else. >>>>> >>>>> As far as I can see only the "mqd per userspace queue" might be a bit >>>>> questionable, everything else sounds reasonable. >>>>> >>>>> Christian. >>>> >>>> Most of the pin downs are done on device initialization. >>>> The "mqd per userspace" is done per userspace queue creation. However, as I >>>> said, it has an upper limit of 128MB on KV, and considering the 2G local >>>> memory, I think it is OK. >>>> The runlists are also done on userspace queue creation/deletion, but we only >>>> have 1 or 2 runlists per device, so it is not that bad. >>> >>> 2G local memory ? You can not assume anything on userside configuration some >>> one might build an hsa computer with 512M and still expect a functioning >>> desktop. >> First of all, I'm only considering Kaveri computer, not "hsa" computer. >> Second, I would imagine we can build some protection around it, like >> checking total local memory and limit number of queues based on some >> percentage of that total local memory. So, if someone will have only >> 512M, he will be able to open less queues. >> >> >>> >>> I need to go look into what all this mqd is for, what it does and what it is >>> about. But pinning is really bad and this is an issue with userspace command >>> scheduling an issue that obviously AMD fails to take into account in design >>> phase. >> Maybe, but that is the H/W design non-the-less. We can't very well >> change the H/W. > > You can not change the hardware but it is not an excuse to allow bad design to > sneak in software to work around that. So i would rather penalize bad hardware > design and have command submission in the kernel, until AMD fix its hardware to > allow proper scheduling by the kernel and proper control by the kernel. I'm sorry but I do *not* think this is a bad design. S/W scheduling in the kernel can not, IMO, scale well to 100K queues and 10K processes. > Because really where we want to go is having GPU closer to a CPU in term of scheduling > capacity and once we get there we want the kernel to always be able to take over > and do whatever it wants behind process back. Who do you refer to when you say "we" ? AFAIK, the hw scheduling direction is where AMD is now and where it is heading in the future. That doesn't preclude the option to allow the kernel to take over and do what he wants. I agree that in KV we have a problem where we can't do a mid-wave preemption, so theoretically, a long running compute kernel can make things messy, but in Carrizo, we will have this ability. Having said that, it will only be through the CP H/W scheduling. So AMD is _not_ going to abandon H/W scheduling. You can dislike it, but this is the situation. > >>>>> >>>>>>> >>>>>>> It might be better to add a drivers/gpu/drm/amd directory and add common >>>>>>> stuff there. >>>>>>> >>>>>>> Given that this is not intended to be final HSA api AFAICT then i would >>>>>>> say this far better to avoid the whole kfd module and add ioctl to radeon. >>>>>>> This would avoid crazy communication btw radeon and kfd. >>>>>>> >>>>>>> The whole aperture business needs some serious explanation. Especialy as >>>>>>> you want to use userspace address there is nothing to prevent userspace >>>>>>> program from allocating things at address you reserve for lds, scratch, >>>>>>> ... only sane way would be to move those lds, scratch inside the virtual >>>>>>> address reserved for kernel (see kernel memory map). >>>>>>> >>>>>>> The whole business of locking performance counter for exclusive per process >>>>>>> access is a big NO. Which leads me to the questionable usefullness of user >>>>>>> space command ring. >>>>>> That's like saying: "Which leads me to the questionable usefulness of HSA". I >>>>>> find it analogous to a situation where a network maintainer nacking a driver >>>>>> for a network card, which is slower than a different network card. Doesn't >>>>>> seem reasonable this situation is would happen. He would still put both the >>>>>> drivers in the kernel because people want to use the H/W and its features. So, >>>>>> I don't think this is a valid reason to NACK the driver. >>> >>> Let me rephrase, drop the the performance counter ioctl and modulo memory pinning >>> i see no objection. In other word, i am not NACKING whole patchset i am NACKING >>> the performance ioctl. >>> >>> Again this is another argument for round trip to the kernel. As inside kernel you >>> could properly do exclusive gpu counter access accross single user cmd buffer >>> execution. >>> >>>>>> >>>>>>> I only see issues with that. First and foremost i would >>>>>>> need to see solid figures that kernel ioctl or syscall has a higher an >>>>>>> overhead that is measurable in any meaning full way against a simple >>>>>>> function call. I know the userspace command ring is a big marketing features >>>>>>> that please ignorant userspace programmer. But really this only brings issues >>>>>>> and for absolutely not upside afaict. >>>>>> Really ? You think that doing a context switch to kernel space, with all its >>>>>> overhead, is _not_ more expansive than just calling a function in userspace >>>>>> which only puts a buffer on a ring and writes a doorbell ? >>> >>> I am saying the overhead is not that big and it probably will not matter in most >>> usecase. For instance i did wrote the most useless kernel module that add two >>> number through an ioctl (http://people.freedesktop.org/~glisse/adder.tar) and >>> it takes ~0.35microseconds with ioctl while function is ~0.025microseconds so >>> ioctl is 13 times slower. >>> >>> Now if there is enough data that shows that a significant percentage of jobs >>> submited to the GPU will take less that 0.35microsecond then yes userspace >>> scheduling does make sense. But so far all we have is handwaving with no data >>> to support any facts. >>> >>> >>> Now if we want to schedule from userspace than you will need to do something >>> about the pinning, something that gives control to kernel so that kernel can >>> unpin when it wants and move object when it wants no matter what userspace is >>> doing. >>> >>>>>>> >>>>>>> So i would rather see a very simple ioctl that write the doorbell and might >>>>>>> do more than that in case of ring/queue overcommit where it would first have >>>>>>> to wait for a free ring/queue to schedule stuff. This would also allow sane >>>>>>> implementation of things like performance counter that could be acquire by >>>>>>> kernel for duration of a job submitted by userspace. While still not optimal >>>>>>> this would be better that userspace locking. >>>>>>> >>>>>>> >>>>>>> I might have more thoughts once i am done with all the patches. >>>>>>> >>>>>>> Cheers, >>>>>>> Jérôme >>>>>>> >>>>>>>> >>>>>>>> Original Cover Letter: >>>>>>>> >>>>>>>> This patch set implements a Heterogeneous System Architecture (HSA) driver >>>>>>>> for radeon-family GPUs. >>>>>>>> HSA allows different processor types (CPUs, DSPs, GPUs, etc..) to share >>>>>>>> system resources more effectively via HW features including shared pageable >>>>>>>> memory, userspace-accessible work queues, and platform-level atomics. In >>>>>>>> addition to the memory protection mechanisms in GPUVM and IOMMUv2, the Sea >>>>>>>> Islands family of GPUs also performs HW-level validation of commands passed >>>>>>>> in through the queues (aka rings). >>>>>>>> >>>>>>>> The code in this patch set is intended to serve both as a sample driver for >>>>>>>> other HSA-compatible hardware devices and as a production driver for >>>>>>>> radeon-family processors. The code is architected to support multiple CPUs >>>>>>>> each with connected GPUs, although the current implementation focuses on a >>>>>>>> single Kaveri/Berlin APU, and works alongside the existing radeon kernel >>>>>>>> graphics driver (kgd). >>>>>>>> AMD GPUs designed for use with HSA (Sea Islands and up) share some hardware >>>>>>>> functionality between HSA compute and regular gfx/compute (memory, >>>>>>>> interrupts, registers), while other functionality has been added >>>>>>>> specifically for HSA compute (hw scheduler for virtualized compute rings). >>>>>>>> All shared hardware is owned by the radeon graphics driver, and an interface >>>>>>>> between kfd and kgd allows the kfd to make use of those shared resources, >>>>>>>> while HSA-specific functionality is managed directly by kfd by submitting >>>>>>>> packets into an HSA-specific command queue (the "HIQ"). >>>>>>>> >>>>>>>> During kfd module initialization a char device node (/dev/kfd) is created >>>>>>>> (surviving until module exit), with ioctls for queue creation & management, >>>>>>>> and data structures are initialized for managing HSA device topology. >>>>>>>> The rest of the initialization is driven by calls from the radeon kgd at the >>>>>>>> following points : >>>>>>>> >>>>>>>> - radeon_init (kfd_init) >>>>>>>> - radeon_exit (kfd_fini) >>>>>>>> - radeon_driver_load_kms (kfd_device_probe, kfd_device_init) >>>>>>>> - radeon_driver_unload_kms (kfd_device_fini) >>>>>>>> >>>>>>>> During the probe and init processing per-device data structures are >>>>>>>> established which connect to the associated graphics kernel driver. This >>>>>>>> information is exposed to userspace via sysfs, along with a version number >>>>>>>> allowing userspace to determine if a topology change has occurred while it >>>>>>>> was reading from sysfs. >>>>>>>> The interface between kfd and kgd also allows the kfd to request buffer >>>>>>>> management services from kgd, and allows kgd to route interrupt requests to >>>>>>>> kfd code since the interrupt block is shared between regular >>>>>>>> graphics/compute and HSA compute subsystems in the GPU. >>>>>>>> >>>>>>>> The kfd code works with an open source usermode library ("libhsakmt") which >>>>>>>> is in the final stages of IP review and should be published in a separate >>>>>>>> repo over the next few days. >>>>>>>> The code operates in one of three modes, selectable via the sched_policy >>>>>>>> module parameter : >>>>>>>> >>>>>>>> - sched_policy=0 uses a hardware scheduler running in the MEC block within >>>>>>>> CP, and allows oversubscription (more queues than HW slots) >>>>>>>> - sched_policy=1 also uses HW scheduling but does not allow >>>>>>>> oversubscription, so create_queue requests fail when we run out of HW slots >>>>>>>> - sched_policy=2 does not use HW scheduling, so the driver manually assigns >>>>>>>> queues to HW slots by programming registers >>>>>>>> >>>>>>>> The "no HW scheduling" option is for debug & new hardware bringup only, so >>>>>>>> has less test coverage than the other options. Default in the current code >>>>>>>> is "HW scheduling without oversubscription" since that is where we have the >>>>>>>> most test coverage but we expect to change the default to "HW scheduling >>>>>>>> with oversubscription" after further testing. This effectively removes the >>>>>>>> HW limit on the number of work queues available to applications. >>>>>>>> >>>>>>>> Programs running on the GPU are associated with an address space through the >>>>>>>> VMID field, which is translated to a unique PASID at access time via a set >>>>>>>> of 16 VMID-to-PASID mapping registers. The available VMIDs (currently 16) >>>>>>>> are partitioned (under control of the radeon kgd) between current >>>>>>>> gfx/compute and HSA compute, with each getting 8 in the current code. The >>>>>>>> VMID-to-PASID mapping registers are updated by the HW scheduler when used, >>>>>>>> and by driver code if HW scheduling is not being used. >>>>>>>> The Sea Islands compute queues use a new "doorbell" mechanism instead of the >>>>>>>> earlier kernel-managed write pointer registers. Doorbells use a separate BAR >>>>>>>> dedicated for this purpose, and pages within the doorbell aperture are >>>>>>>> mapped to userspace (each page mapped to only one user address space). >>>>>>>> Writes to the doorbell aperture are intercepted by GPU hardware, allowing >>>>>>>> userspace code to safely manage work queues (rings) without requiring a >>>>>>>> kernel call for every ring update. >>>>>>>> First step for an application process is to open the kfd device. Calls to >>>>>>>> open create a kfd "process" structure only for the first thread of the >>>>>>>> process. Subsequent open calls are checked to see if they are from processes >>>>>>>> using the same mm_struct and, if so, don't do anything. The kfd per-process >>>>>>>> data lives as long as the mm_struct exists. Each mm_struct is associated >>>>>>>> with a unique PASID, allowing the IOMMUv2 to make userspace process memory >>>>>>>> accessible to the GPU. >>>>>>>> Next step is for the application to collect topology information via sysfs. >>>>>>>> This gives userspace enough information to be able to identify specific >>>>>>>> nodes (processors) in subsequent queue management calls. Application >>>>>>>> processes can create queues on multiple processors, and processors support >>>>>>>> queues from multiple processes. >>>>>>>> At this point the application can create work queues in userspace memory and >>>>>>>> pass them through the usermode library to kfd to have them mapped onto HW >>>>>>>> queue slots so that commands written to the queues can be executed by the >>>>>>>> GPU. Queue operations specify a processor node, and so the bulk of this code >>>>>>>> is device-specific. >>>>>>>> Written by John Bridgman >>>>>>>> >>>>>>>> >>>>>>>> Alexey Skidanov (1): >>>>>>>> amdkfd: Implement the Get Process Aperture IOCTL >>>>>>>> >>>>>>>> Andrew Lewycky (3): >>>>>>>> amdkfd: Add basic modules to amdkfd >>>>>>>> amdkfd: Add interrupt handling module >>>>>>>> amdkfd: Implement the Set Memory Policy IOCTL >>>>>>>> >>>>>>>> Ben Goz (8): >>>>>>>> amdkfd: Add queue module >>>>>>>> amdkfd: Add mqd_manager module >>>>>>>> amdkfd: Add kernel queue module >>>>>>>> amdkfd: Add module parameter of scheduling policy >>>>>>>> amdkfd: Add packet manager module >>>>>>>> amdkfd: Add process queue manager module >>>>>>>> amdkfd: Add device queue manager module >>>>>>>> amdkfd: Implement the create/destroy/update queue IOCTLs >>>>>>>> >>>>>>>> Evgeny Pinchuk (3): >>>>>>>> amdkfd: Add topology module to amdkfd >>>>>>>> amdkfd: Implement the Get Clock Counters IOCTL >>>>>>>> amdkfd: Implement the PMC Acquire/Release IOCTLs >>>>>>>> >>>>>>>> Oded Gabbay (10): >>>>>>>> mm: Add kfd_process pointer to mm_struct >>>>>>>> drm/radeon: reduce number of free VMIDs and pipes in KV >>>>>>>> drm/radeon/cik: Don't touch int of pipes 1-7 >>>>>>>> drm/radeon: Report doorbell configuration to amdkfd >>>>>>>> drm/radeon: adding synchronization for GRBM GFX >>>>>>>> drm/radeon: Add radeon <--> amdkfd interface >>>>>>>> Update MAINTAINERS and CREDITS files with amdkfd info >>>>>>>> amdkfd: Add IOCTL set definitions of amdkfd >>>>>>>> amdkfd: Add amdkfd skeleton driver >>>>>>>> amdkfd: Add binding/unbinding calls to amd_iommu driver >>>>>>>> >>>>>>>> CREDITS | 7 + >>>>>>>> MAINTAINERS | 10 + >>>>>>>> drivers/gpu/drm/radeon/Kconfig | 2 + >>>>>>>> drivers/gpu/drm/radeon/Makefile | 3 + >>>>>>>> drivers/gpu/drm/radeon/amdkfd/Kconfig | 10 + >>>>>>>> drivers/gpu/drm/radeon/amdkfd/Makefile | 14 + >>>>>>>> drivers/gpu/drm/radeon/amdkfd/cik_mqds.h | 185 +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/cik_regs.h | 220 ++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_aperture.c | 123 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_chardev.c | 518 +++++++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_crat.h | 294 +++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_device.c | 254 ++++ >>>>>>>> .../drm/radeon/amdkfd/kfd_device_queue_manager.c | 985 ++++++++++++++++ >>>>>>>> .../drm/radeon/amdkfd/kfd_device_queue_manager.h | 101 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_doorbell.c | 264 +++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_interrupt.c | 161 +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.c | 305 +++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.h | 66 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_module.c | 131 +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.c | 291 +++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.h | 54 + >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_packet_manager.c | 488 ++++++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pasid.c | 97 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pm4_headers.h | 682 +++++++++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_pm4_opcodes.h | 107 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_priv.h | 466 ++++++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_process.c | 405 +++++++ >>>>>>>> .../drm/radeon/amdkfd/kfd_process_queue_manager.c | 343 ++++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_queue.c | 109 ++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_topology.c | 1207 >>>>>>>> ++++++++++++++++++++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_topology.h | 168 +++ >>>>>>>> drivers/gpu/drm/radeon/amdkfd/kfd_vidmem.c | 96 ++ >>>>>>>> drivers/gpu/drm/radeon/cik.c | 154 +-- >>>>>>>> drivers/gpu/drm/radeon/cik_reg.h | 65 ++ >>>>>>>> drivers/gpu/drm/radeon/cikd.h | 51 +- >>>>>>>> drivers/gpu/drm/radeon/radeon.h | 9 + >>>>>>>> drivers/gpu/drm/radeon/radeon_device.c | 32 + >>>>>>>> drivers/gpu/drm/radeon/radeon_drv.c | 5 + >>>>>>>> drivers/gpu/drm/radeon/radeon_kfd.c | 566 +++++++++ >>>>>>>> drivers/gpu/drm/radeon/radeon_kfd.h | 119 ++ >>>>>>>> drivers/gpu/drm/radeon/radeon_kms.c | 7 + >>>>>>>> include/linux/mm_types.h | 14 + >>>>>>>> include/uapi/linux/kfd_ioctl.h | 133 +++ >>>>>>>> 43 files changed, 9226 insertions(+), 95 deletions(-) >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/Kconfig >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/Makefile >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/cik_mqds.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/cik_regs.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_aperture.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_chardev.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_crat.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device_queue_manager.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_device_queue_manager.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_doorbell.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_interrupt.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_kernel_queue.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_module.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_mqd_manager.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_packet_manager.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pasid.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pm4_headers.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_pm4_opcodes.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_priv.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_process.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_process_queue_manager.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_queue.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_topology.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_topology.h >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/amdkfd/kfd_vidmem.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/radeon_kfd.c >>>>>>>> create mode 100644 drivers/gpu/drm/radeon/radeon_kfd.h >>>>>>>> create mode 100644 include/uapi/linux/kfd_ioctl.h >>>>>>>> >>>>>>>> -- >>>>>>>> 1.9.1 >>>>>>>> >>>>>> >>>>> >>>> >>