From: Michael Neuling <mikey@neuling.org>
To: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>,
Michael Ellerman <mpe@ellerman.id.au>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>,
stewart@linux.vnet.ibm.com, apopple@au1.ibm.com,
hbabu@us.ibm.com, oohall@gmail.com, linuxppc-dev@ozlabs.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v6 17/17] powerpc/vas: Document FTW API/usage
Date: Mon, 14 Aug 2017 20:43:30 +1000 [thread overview]
Message-ID: <1502707410.9844.10.camel@neuling.org> (raw)
In-Reply-To: <1502233622-9330-18-git-send-email-sukadev@linux.vnet.ibm.com>
T24gVHVlLCAyMDE3LTA4LTA4IGF0IDE2OjA3IC0wNzAwLCBTdWthZGV2IEJoYXR0aXByb2x1IHdy
b3RlOgo+IERvY3VtZW50IHRoZSB1c2FnZSBvZiB0aGUgVkFTIEZhc3QgdGhyZWFkLXdha2V1cCBB
UEkuCj4gCj4gVGhhbmtzIGZvciBpbnB1dC9jb21tZW50cyBmcm9tIEJlbmphbWluIEhlcnJlbnNj
aG1pZHQsIE1pY2hhZWwgTmV1bGluZywKPiBNaWNoYWVsIEVsbGVybWFuLCBSb2JlcnQgQmxhY2tt
b3JlLCBJYW4gTXVuc2llLCBIYXJlbiBNeW5lbmksIFBhdWwgTWFja2VycmFzLgo+IAo+IENjOklh
biBNdW5zaWUgPGltdW5zaWVAYXUxLmlibS5jb20+Cj4gQ2M6UGF1bCBNYWNrZXJyYXMgPHBhdWx1
c0BvemxhYnMub3JnPgo+IFNpZ25lZC1vZmYtYnk6IFN1a2FkZXYgQmhhdHRpcHJvbHUgPHN1a2Fk
ZXZAbGludXgudm5ldC5pYm0uY29tPgo+IC0tLQo+IMKgRG9jdW1lbnRhdGlvbi9wb3dlcnBjL2Z0
dy1hcGkudHh0IHwgMzczCj4gKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysK
PiDCoDEgZmlsZSBjaGFuZ2VkLCAzNzMgaW5zZXJ0aW9ucygrKQo+IMKgY3JlYXRlIG1vZGUgMTAw
NjQ0IERvY3VtZW50YXRpb24vcG93ZXJwYy9mdHctYXBpLnR4dAo+IAo+IGRpZmYgLS1naXQgYS9E
b2N1bWVudGF0aW9uL3Bvd2VycGMvZnR3LWFwaS50eHQgYi9Eb2N1bWVudGF0aW9uL3Bvd2VycGMv
ZnR3LQo+IGFwaS50eHQKPiBuZXcgZmlsZSBtb2RlIDEwMDY0NAo+IGluZGV4IDAwMDAwMDAuLjBi
M2YxNmYKPiAtLS0gL2Rldi9udWxsCj4gKysrIGIvRG9jdW1lbnRhdGlvbi9wb3dlcnBjL2Z0dy1h
cGkudHh0Cj4gQEAgLTAsMCArMSwzNzMgQEAKPiArVmlydHVhbCBBY2NlbGVyYXRvciBTd2l0Y2hi
b2FyZCBhbmQgRmFzdCBUaHJlYWQtV2FrZXVwIEFQSQo+ICsKPiArwqDCoMKgwqBQb3dlcjkgcHJv
Y2Vzc29yIHN1cHBvcnRzIGEgaGFyZHdhcmUgc3VieXN0ZW0ga25vd24gYXMgdGhlIFZpcnR1YWwK
PiArwqDCoMKgwqBBY2NlbGVyYXRvciBTd2l0Y2hib2FyZCAoVkFTKSB3aGljaCBhbGxvd3MgdHdv
IGVudGl0aWVzIGluIHRoZSBQb3dlcjkKPiArwqDCoMKgwqBzeXN0ZW0gdG8gZWZmaWNpZW50bHkg
ZXhjaGFuZ2UgbWVzc2FnZXMuIE1lc3NhZ2VzIG11c3QgYmUgZm9ybWF0dGVkIGFzCj4gK8KgwqDC
oMKgQ29wcm9jZXNzb3IgUmVxZXVzdCBCbG9ja3MgKENSQikgYW5kIGJlIHN1Ym1pdHRlZCB1c2lu
ZyB0aGUgQ09QWS9QQVNURQo+ICvCoMKgwqDCoGluc3RydWN0aW9ucyAobmV3IGluIFBvd2VyOSku
Cj4gKwo+ICvCoMKgwqDCoFVzYWdlIG9mIFZBUyBkZXBlbmRzIG9uIHRoZSBlbnRpdGllcyBleGNo
YW5naW5nIHRoZSBtZXNzYWdlcyBhbmQKPiArwqDCoMKgwqBjdXJyZW50bHkgdHdvIHVzYWdlcyBo
YXZlIGJlZW4gaWRlbnRpZmllZC4KPiArCj4gK8KgwqDCoMKgRmlyc3QgdXNhZ2Ugb2YgVkFTLCBy
ZWZlcnJlZCB0byBhcyBWQVMvTlggaW52b2x2ZXMgYSBzb2Z0d2FyZSB0aHJlYWQKPiArwqDCoMKg
wqBzdWJtaXR0aW5nIGRhdGEgY29tcHJlc3Npb24gcmVxdWVzdHMgdG8gYSBjby1wcm9jZXNzb3Ig
KGhhcmR3YXJlL25lc3QKPiArwqDCoMKgwqBhY2NlbGVyYXRvcikgYWthIE5YIGVuZ2luZS4gVGhl
IEFQSSBmb3IgdGhpcyB1c2FnZSBpcyBkZXNjcmliZWQgaW4gdGhlCj4gK8KgwqDCoMKgVkFTL05Y
IEFQSSBkb2N1bWVudC4KPiArCj4gK8KgwqDCoMKgQWx0ZXJuYXRpdmVseSwgVkFTIGNhbiBiZSB1
c2VkIGJ5IHR3byBzb2Z0d2FyZSB0aHJlYWRzIHRvIGVmZmljaWVudGx5Cj4gK8KgwqDCoMKgZXhj
aGFuZ2UgbWVzc2FnZXMuIEluaXRpYWxseSwgdGhpcyBtZWNoYW5pc20gaXMgaW50ZW5kZWQgdG8g
d2FrZSB1cCBhCj4gK8KgwqDCoMKgd2FpdGluZyB0aHJlYWQgcXVpY2tseSAtIGkuZSAiZmFzdCB0
aHJlYWQgd2FrZS11cCAoRlRXKSIuIFRoaXMgZG9jdW1lbnQKPiArwqDCoMKgwqBkZXNjcmliZXMg
dGhlIHVzZXIgQVBJIGZvciB0aGlzIFZBUy9GVFcgbWVjaGFuaXNtLgo+ICsKPiArwqDCoMKgwqBB
cHBsaWNhdGlvbiBhY2Nlc3MgdG8gdGhlIEZUVyBtZWNoYW5pc20gaXMgcHJvdmlkZWQgdGhyb3Vn
aCB0aGUgTlgtRlRXCj4gK8KgwqDCoMKgZGV2aWNlIG5vZGUgKC9kZXYvY3J5cHRvL254LWZ0dykg
aW1wbGVtZW50ZWQgYnkgdGhlIFZBUy9GVFcgZGV2aWNlCj4gK8KgwqDCoMKgZHJpdmVyLgoKY3J5
cHRvPwoKPiArCj4gK8KgwqDCoMKgQSBzb2Z0d2FyZSB0aHJlYWQgVDEgdGhhdCBpbnRlbmRzIHRv
IHdhaXQgZm9yIGFuIGV2ZW50IG11c3QgZmlyc3Qgc2V0dXAKPiArwqDCoMKgwqBhIHJlY2VpdmUg
d2luZG93LCBieSBvcGVuaW5nIHRoZSBOWC1GVFcgZGV2aWNlIGFuZCB1c2luZyB0aGUKPiArwqDC
oMKgwqBWQVNfUlhfV0lOX09QRU4gaW9jdGwuIFVwb24gc3VjY2Vzc2Z1bCByZXR1cm4gZnJvbSB0
aGUgVkFTX1JYX1dJTl9PUEVOCj4gK8KgwqDCoMKgaW9jdGwsIGFuIHJ4X3dpbl9oYW5kbGUgaXMg
cmV0dXJuZWQuCgpJIHJlYWxpc2UgdGhlcmUgaXMgYSB3aW5kb3cgaGVyZSBhcyBwYXJ0IG9mIHRo
ZSBoYXJkd2FyZSBpbXBsZW1lbnRhdGlvbiwgYnV0IHRoZQp1c2VycyBkb24ndCBjYXJlIGFib3V0
IHRoZSB3aW5kb3cgb24gdGhlIHJlY2VpdmUgc2lkZS4gSXQncyBoaWRkZW4gZnJvbSB0aGVtLiAK
SXQncyBqdXN0IGFuIHJ4IGhhbmRsZSBJTUhPLgoKVGhlIHNlbmRlciBjZXJ0YWlubHkgaGFzIGEg
d2luZG93IHRoYXQgdXNlcnMgY2FyZSBhYm91dCBzaW5jZSB0aGV5IGhhdmUgdG8gbW1hcAppdC4K
Cj4gKwo+ICvCoMKgwqDCoEEgc29mdHdhcmUgdGhyZWFkIFQyIHRoYXQgaW50ZW5kcyB0byB3YWtl
IHVwIFQxIGF0IHNvbWUgcG9pbnQsIG11c3QgZmlyc3QKPiArwqDCoMKgwqBzZXQgdXAgYSAic2Vu
ZCB3aW5kb3ciIHVzaW5nIHRoZSBWQVNfVFhfV0lOX09QRU4gaW9jdGwgYW5kIHNwZWNpZnkgdGhl
Cj4gK8KgwqDCoMKgcnhfd2luX2hhbmRsZSBvYnRhaW5lZCBieSBUMS4gQWZ0ZXIgYSBzdWNjZXNz
ZnVsIFZBU19UWF9XSU5fT1BFTiBpb2N0bAo+IHRoZQo+ICvCoMKgwqDCoHNlbmQgd2luZG93IG9m
IFQyIGlzIGNvbnNpZGVyZWQgcGFpcmVkIHdpdGggdGhlIHJlY2VpdmUgd2luZG93IG9mIFQxLiBU
aGUKPiArwqDCoMKgwqB0aHJlYWQgVDIgbXVzdCB0aGVuIHVzZSBtbWFwKCkgdG8gb2J0YWluIGEg
InBhc3RlIGFkZHJlc3MiIGZvciB0aGUgc2VuZAo+ICvCoMKgwqDCoHdpbmRvdy4KCgo+ICvCoMKg
wqDCoFdpdGggdGhpcyBzZXQgdXAsIHRocmVhZCBUMSBjYW4gd2FpdCBmb3IgYW4gZXZlbnQgdXNp
bmcgdGhlIFdBSVQKPiArwqDCoMKgwqBpbnN0cnVjdGlvbi4KPiArCj4gK8KgwqDCoMKgVGhyZWFk
IFQyIGNhbiB3YWtlIHVwIFQxIGJ5IHVzaW5nIHRoZSAiQ09QWS9QQVNURSIgaW5zdHJ1Y3Rpb25z
IGFuZAo+ICvCoMKgwqDCoHN1Ym1pdHRpbmcgYW4gZW1wdHkvTlVMTCBDUkIgdG8gdGhlIHNlbmQg
d2luZG93J3MgcGFzdGUgYWRkcmVzcy4gVGhlCj4gK8KgwqDCoMKgd2FpdC93YWtlIHVwIHByb2Nl
c3MgY2FuIGJlIHJlcGVhdGVkIGFzIGxvbmcgYXMgdGhlIHRocmVhZHMgaGF2ZSB0aGUKPiArwqDC
oMKgwqBzZW5kL3JlY2VpdmUgd2luZG93cyBvcGVuLgoKCgo+ICsxLiBOWC1GVFcgRGV2aWNlIE5v
ZGUKPiArCj4gK8KgwqDCoMKgVGhlcmUgaXMgb25lIC9kZXYvY3J5cHRvL254LWZ0dyBub2RlIGlu
IHRoZSBzeXN0ZW0gYW5kIGl0IHByb3ZpZGVzCj4gK8KgwqDCoMKgYWNjZXNzIHRvIHRoZSBWQVMv
RlRXIGZ1bmN0aW9uYWxpdHkuCgoKPiArwqDCoMKgwqBUaGUgb25seSB2YWxpZCBvcGVyYXRpb25z
IG9uIHRoZSBOWC1GVFcgbm9kZSBhcmU6Cj4gKwo+ICvCoMKgwqDCoMKgwqDCoMKgLSBvcGVuKCkg
dGhlIGRldmljZSBmb3IgcmVhZCBhbmQgd3JpdGUuCj4gKwo+ICvCoMKgwqDCoMKgwqDCoMKgLSBp
c3N1ZSBlaXRoZXIgVkFTX1JYX1dJTl9PUEVOIG9yIFZBU19UWF9XSU5fT1BFTiBpb2N0bHMgdG8g
c2V0IHVwCj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgcmVjZWl2ZSBvciBzZW5kIChvbmx5IG9uZSBv
ZiB0aGVtIHBlciBvcGVuKS4KPiArCj4gK8KgwqDCoMKgwqDCoMKgwqAtIGlmIHRoZSBvcGVuIGlz
IGFzc29jaWF0ZWQgd2l0aCBzZW5kIHdpbmRvdyAoaS5lIFZBU19UWF9XSU5fT1BFTgo+ICvCoMKg
wqDCoMKgwqDCoMKgwqDCoGlvY3RsIHdhcyBpc3N1ZWQpIG1tYXAoKSB0aGUgc2VuZCB3aW5kb3cg
aW50byB0aGUgYXBwbGljYXRpb24ncwo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoHZpcnR1YWwgYWRk
cmVzcyBzcGFjZS4gKGkuZSBnZXQgYSAncGFzdGVfYWRkcmVzcycgZm9yIHRoZSBzZW5kCj4gK8Kg
wqDCoMKgwqDCoMKgwqDCoMKgd2luZG93KS4KPiArCj4gK8KgwqDCoMKgwqDCoMKgwqAtIGNsb3Nl
IHRoZSBkZXZpY2Ugbm9kZS4KPiArCj4gK8KgwqDCoMKgT3RoZXIgZmlsZSBvcGVyYXRpb25zIG9u
IHRoZSBOWC1GVFcgbm9kZSBhcmUgdW5kZWZpbmVkLgo+ICsKPiArwqDCoMKgwqBOb3RlIHRIQVQg
dGhlIENPUFkgYW5kIFBBU1RFIG9wZXJhdGlvbnMgZ28gZGlyZWN0bHkgdG8gdGhlIGhhcmR3YXJl
Cj4gK8KgwqDCoMKgYW5kIG5vdCBnbyB0aHJvdWdoIHRoZSBOWC1GVFcgZGV2aWNlLgoKSSBkb24n
dCB1bmRlcnN0YW5kIHRoaXMgc3RhdGVtZW50Cgo+ICsKPiArwqDCoMKgwqBBbHRob3VnaCBhIHN5
c3RlbSBtYXkgaGF2ZSBzZXZlcmFsIGluc3RhbmNlcyBvZiB0aGUgVkFTIGluIHRoZSBzeXN0ZW0K
PiArwqDCoMKgwqAodHlwaWNhbGx5LCBvbmUgcGVyIFA5IGNoaXApIHRoZXJlIGlzIGp1c3Qgb25l
IE5YLUZUVyBkZXZpY2Ugbm9kZSBpbgo+ICvCoMKgwqDCoHRoZSBzeXN0ZW0uCgo+ICsJV2hlbiB0
aGUgTlgtRlRXIGRldmljZSBub2RlIGlzIG9wZW5lZCwgdGhlIGtlcm5lbCBhc3NpZ25zIGEgc3Vp
dGFibGUKPiArCWluc3RhbmNlIG9mIFZBUyB0byB0aGUgcHJvY2Vzcy4gS2VybmVsIHdpbGwgbWFr
ZSBhIGJlc3QtZWZmb3J0Cj4gYXR0ZW1wdAo+ICsJdG8gYXNzaWduIGFuIG9wdGltYWwgaW5zdGFu
Y2Ugb2YgVkFTIGZvciB0aGUgcHJvY2Vzcy4gSW4gdGhlIGluaXRpYWwKPiArwqDCoMKgwqByZWxl
YXNlLCB0aGUga2VybmVsIGRvZXMgbm90IHN1cHBvcnQgbWlncmF0aW5nIHRoZSBWQVMgaW5zdGFu
Y2UgaWYgdGhlCj4gK8KgwqDCoMKgcHJvY2VzcyBtaWdyYXRlcyBmcm9tIGEgcHJvY2Vzc29yIG9u
IG9uZSBjaGlwIHRvIGEgcHJvY2Vzc29yIG9uIGFub3RoZXIKPiArwqDCoMKgwqBjaGlwLgoKSG93
IGlzIGl0ICJvcHRpbWFsIj8KCj4gK8KgwqDCoMKgQXBwbGljYXRpb25zIG1heSBjaG9zZSBhIHNw
ZWNpZmljIGluc3RhbmNlIG9mIHRoZSBWQVMgdXNpbmcgdGhlICd2YXNfaWQnCj4gK8KgwqDCoMKg
ZmllbGQgaW4gdGhlIFZBU19UWF9XSU5fT1BFTiBhbmQgVkFTX1JYX1dJTl9PUEVOIGlvY3RscyBh
cyBkZXRhaWxlZAo+IGJlbG93LgoKCgoKPiArMi4gT3BlbiBOWC1GVFcgbm9kZQo+ICsKPiArwqDC
oMKgwqBUaGUgZGV2aWNlIHNob3VsZCBiZSBvcGVuZWQgZm9yIHJlYWQgYW5kIHdyaXRlLiBObyBz
cGVjaWFsIHByaXZpbGVnZXMKPiArwqDCoMKgwqBhcmUgbmVlZGVkIHRvIG9wZW4gdGhlIGRldmlj
ZS4gVGhlIGRldmljZSBtYXkgYmUgb3BlbmVkIG11bHRpcGxlIHRpbWVzLgo+ICsKPiArwqDCoMKg
wqBFYWNoIG9wZW4oKSBvZiB0aGUgTlgtRlRXIGRldmljZSBtYXkgYmUgYXNzb2NpYXRlZCB3aXRo
IGVpdGhlciBhIHNlbmQKPiArwqDCoMKgwqB3aW5kb3cgb3IgcmVjZWl2ZSB3aW5kb3cgYnV0IG5v
dCBib3RoLgo+ICsKPiArwqDCoMKgwqBTZWUgb3BlbigyKSBzeXN0ZW0gY2FsbCBtYW4gcGFnZXMg
Zm9yIG90aGVyIGRldGFpbHMgc3VjaCBhcyByZXR1cm4KPiArwqDCoMKgwqB2YWx1ZXMsIGVycm9y
IGNvZGVzIGFuZCByZXN0cmljdGlvbnMuCj4gKwo+ICszLiBTZXR1cCBSZWNlaXZlIHdpbmRvdyAo
VkFTX1JYX1dJTl9PUEVOIGlvY3RsKQo+ICsKPiArwqDCoMKgwqBBIHRocmVhZCB0aGF0IGV4cGVj
dHMgdG8gd2FpdCBmb3IgZXZlbnRzIGFuZCBiZSB3b2tlbiB1cCB1c2luZyBDT1BZL1BBU1RFCj4g
K8KgwqDCoMKgbXVzdCBmaXJzdCBzZXQgdXAgYSByZWNlaXZlIHdpbmRvdyBieSBpc3N1aW5nIHRo
ZSBWQVNfUlhfV0lOX09QRU4gaW9jdGwuCj4gKwo+ICvCoMKgwqDCoMKgwqDCoMKgI2luY2x1ZGUg
PGFzbS92YXMuaD4KPiArCj4gK8KgwqDCoMKgwqDCoMKgwqBzdHJ1Y3QgdmFzX3J4X3dpbl9vcGVu
X2F0dHIgcnhhdHRyOwo+ICsKPiArwqDCoMKgwqDCoMKgwqDCoHJjID0gaW9jdGwoZmQsIFZBU19S
WF9XSU5fT1BFTiwgJnJ4YXR0cik7Cj4gKwo+ICvCoMKgwqDCoFRoZSBhdHRyaWJ1dGVzIG9mIHJ4
YXR0ciBhcmUgYXMgZm9sbG93czoKPiArCj4gK8KgwqDCoMKgwqDCoMKgwqBzdHJ1Y3QgdmFzX3J4
X3dpbl9vcGVuX2F0dHIgewo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoGludDE2
X3TCoMKgwqDCoMKgwqDCoHZlcnNpb247Cj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC
oMKgaW50MTZfdMKgwqDCoMKgwqDCoMKgdmFzX2lkOwo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoMKgwqDCoGludDMyX3TCoMKgwqDCoMKgwqDCoHJ4X3dpbl9oYW5kbGU7wqDCoMKgwqAvKiBv
dXRwdXQgZmllbGQgKi8KPiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqBpbnQ2NF90
wqDCoMKgwqDCoMKgwqByZXNlcnZlZFs4XTsKPiArwqDCoMKgwqDCoMKgwqDCoH07Cj4gKwo+ICvC
oMKgwqDCoFRoZSB2ZXJzaW9uIGZpZWxkIGlkZW50aWZpZXMgdGhlIHZlcnNpb24gb2YgdGhlIEFQ
SSBhbmQgbXVzdCBjdXJyZW50bHkKPiArwqDCoMKgwqBiZSBzZXQgdG8gMS4KPiArCj4gK8KgwqDC
oMKgVGhlIHZhc19pZCBmaWVsZCBpZGVudGlmaWVzIGEgc3BlY2lmaWMgaW5zdGFuY2Ugb2YgdGhl
IFZBUyB0aGF0IHRoZQo+ICvCoMKgwqDCoGFwcGxpY2F0aW9uIHdpc2hlcyB0byBhY2Nlc3MuIFNl
ZSBzZWN0aW9uIG9uIFZBUyBJRCBiZWxvdy4KPiArCj4gK8KgwqDCoMKgVGhlIHJlc2VydmVkIGZp
ZWxkIG11c3QgYmUgc2V0IHRvIGFsbCB6ZXJvZXMuCj4gKwo+ICvCoMKgwqDCoFVwb24gc3VjY2Vz
c2Z1bCByZXR1cm4gZnJvbSB0aGUgaW9jdGwsIHRoZSByeF93aW5faGFuZGxlIGZpZWxkIGNvbnRh
aW5zCj4gK8KgwqDCoMKgYW4gaWRlbnRpZmllciBmb3IgdGhlIFZBUyB3aW5kb3cgYXNzb2NpYXRl
ZCB3aXRoIHRoaXMgInNsZWVwaW5nIiB0aHJlYWQuCj4gKwo+ICvCoMKgwqDCoFRoaXMgcnhfd2lu
X2hhbmRsZSBmaWVsZCBpcyB1c2VkIHRvICJwYWlyIiB0aGlzIHJlY2VpdmUgd2luZG93IHdpdGgg
YQo+ICvCoMKgwqDCoHNlbmQgd2luZG93IGFuZCBtdXN0IGJlIHNwZWNpZmllZCB3aGVuIG9wZW5p
bmcgdGhlIGNvcnJlc3BvbmRpbmcgc2VuZAo+ICvCoMKgwqDCoHdpbmRvdyAoc2VlIHN0cnVjdCB2
YXNfdHhfd2luX29wZW5fYXR0ciBiZWxvdykuCj4gKwo+ICvCoMKgwqDCoFJldHVybiB2YWx1ZToK
PiArCj4gK8KgwqDCoMKgVGhlIFZBU19SWF9XSU5fT1BFTiBpb2N0bCByZXR1cm5zIDAgb24gc3Vj
Y2Vzcy4gT24gZXJyb3IsIGl0IHJldHVybnMgLTEKPiArwqDCoMKgwqBhbmQgc2V0cyB0aGUgZXJy
bm8gdmFyaWFibGUgdG8gaW5kaWNhdGUgdGhlIGVycm9yLgo+ICsKPiArwqDCoMKgwqBFcnJvciBj
b2RlczoKPiArCj4gK8KgwqDCoMKgwqDCoMKgwqBFSU5WQUzCoMKgwqDCoMKgwqB2ZXJzaW9uIGlz
IGludmFsaWQKPiArCj4gK8KgwqDCoMKgwqDCoMKgwqBFSU5WQUzCoMKgwqDCoMKgwqB2YXNfaWQg
aXMgaW52YWxpZAo+ICsKPiArwqDCoMKgwqDCoMKgwqDCoEVJTlZBTMKgwqDCoMKgwqDCoHJlc2Vy
dmVkIGZpZWxkIGlzIG5vdCBzZXQgdG8gemVyb2VzCj4gKwo+ICvCoMKgwqDCoMKgwqDCoMKgRUlO
VkFMwqDCoMKgwqDCoMKgZmQgaXMgYWxyZWFkeSBhc3NvY2lhdGVkIHdpdGggYSBzZW5kIHdpbmRv
dwo+ICsKPiArCj4gKzMuIFNldCB1cCBhIFNlbmQgd2luZG93IChWQVNfVFhfV0lOX09QRU4gaW9j
dGwpCj4gKwo+ICvCoMKgwqDCoEFuIGFwcGxpY2F0aW9uIHRocmVhZCB0aGF0IGV4cGVjdHMgdG8g
d2FrZSB1cCBhIHdhaXRpbmcgdGhyZWFkIHVzaW5nCj4gK8KgwqDCoMKgY29weS9wYXN0ZSwgbXVz
dCBmaXJzdCBzZXQgdXAgYSBzZW5kIHdpbmRvdyB0aGF0IGlzIHBhaXJlZCB3aXRoIHRoZQo+ICvC
oMKgwqDCoHJlY2VpdmUgd2luZG93IG9mIHRoZSB3YWl0aW5nIHRocmVhZC4gVGhpcyBpcyBhY2Nv
bXBsaXNoZWQgdXNpbmcgdGhlCj4gK8KgwqDCoMKgVkFTX1RYX1dJTl9PUEVOIGlvY3RsLgo+ICsK
PiArwqDCoMKgwqDCoMKgwqDCoCNpbmNsdWRlIDxhc20vdmFzLmg+Cj4gKwo+ICvCoMKgwqDCoMKg
wqDCoMKgc3RydWN0IHZhc190eF93aW5fb3Blbl9hdHRyIHR4YXR0cjsKPiArCj4gK8KgwqDCoMKg
wqDCoMKgwqByYyA9IGlvY3RsKGZkLCBWQVNfVFhfV0lOX09QRU4sICZ0eGF0dHIpOwoKU28gd2Ug
dGFsa2VkIGFib3V0IHRoaXMgb2ZmbGluZSBiZWZvcmUuLi4uIHRoZSBmZCBoZXJlIHNob3VsZCBu
b3QgYmUgZnJvbSB0aGUKL2RldiBkZXZpY2UgYnV0IHNob3VsZCBiZSB0aGUgZmQgZnJvbSByeF93
aW5fb3BlbiBpb2N0bC4gCgpBcyB5b3UgaGF2ZSBpdCBoZXJlIHlvdSBwYXNzIHRoZSBoYW5kbGUg
aW4gYXMgYSBwYXJhbWV0ZXIgb2YgaW9jdGwuICBUaGlzIG1lYW5zCmFsbCB0aGUgcGVybWlzc2lv
bnMgY2hlY2tzIGhhdmUgdG8gYmUgZG9uZSBieSB5b3UgYXMgdG8gaWYgdGhlc2UgdHdvIHdpbmRv
d3MgY2FuCmJlIGxpbmtlZC4gIElmIHlvdSB1c2UgdGhlIGZkIGZyb20gYmVmb3JlLCB5b3UgY2Fu
IGFzc3VtZSBpZiB0aGUgcmVjZWl2ZXIgaGFzCmdpdmVuIHRoaXMgZmQgdG8gdGhlIHNlbmRlciwg
aXQgaGFzIHRoZSByaWdodCBwZXJtaXNzaW9ucy4KCkkgaGF2ZSBzb21lIHBzZXVkbyBjb2RlIGF0
IHRoZSBlbmQgc2hvd3MgdGhpcy4KCj4gK8KgwqDCoMKgVGhlIGF0dHJpYnV0ZXMgJ3R4YXR0cicg
Zm9yIHRoZSBWQVNfVFhfV0lOX09QRU4gaW9jdGwgYXJlIGRlZmluZWQgYXMKPiArwqDCoMKgwqBm
b2xsb3dzOgo+ICsKPiArwqDCoMKgwqDCoMKgwqDCoHN0cnVjdCB2YXNfdHhfd2luX29wZW5fYXR0
ciB7Cj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoGludDMyX3TCoMKgwqDCoMKgwqDCoHZlcnNp
b247Cj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoGludDE2X3TCoMKgwqDCoMKgwqDCoHZhc19p
ZDsKPiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgdWludDMyX3TCoMKgwqDCoMKgwqByeF93aW5f
aGFuZGxlOwo+ICsKPiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgaW50NjRfdMKgwqDCoMKgwqDC
oMKgcmVzZXJ2ZWQxOwo+ICsKPiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgaW50NjRfdMKgwqDC
oMKgwqDCoMKgZmxhZ3M7Cj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoGludDY0X3TCoMKgwqDC
oMKgwqDCoHJlc2VydmVkMjsKPiArCj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoGludDMyX3TC
oMKgwqDCoMKgwqDCoHRjX21vZGU7Cj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoGludDMyX3TC
oMKgwqDCoMKgwqDCoHJzdmRfdHhidWY7Cj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoGludDY0
X3TCoMKgwqDCoMKgwqDCoHJlc2VydmVkM1s2XTsKPiArwqDCoMKgwqDCoMKgwqDCoH07Cj4gKwo+
ICvCoMKgwqDCoFRoZSB2ZXJzaW9uIGZpZWxkIG11c3QgY3VycmVudGx5IGJlIHNldCB0byAxLgo+
ICsKPiArwqDCoMKgwqBUaGUgdmFzX2lkIGZpZWxkIGlkZW50aWZpZXMgYSBzcGVjaWZpYyBpbnN0
YW5jZSBvZiB0aGUgVkFTIHRoYXQgdGhlCj4gK8KgwqDCoMKgYXBwbGljYXRpb24gd2lzaGVzIHRv
IGFjY2Vzcy4gU2VlIHNlY3Rpb24gb24gVkFTIElEIGJlbG93LgoKQ2FuIHRoaXMgYmUgZGlmZmVy
ZW50IHRvIHRoZSByeD8KCj4gK8KgwqDCoMKgVGhlIHJ4X3dpbl9oYW5kbGUgZmllbGQgbXVzdCBi
ZSBzZXQgdG8gdGhlIHJ4X3dpbl9oYW5kbGUgcmV0dXJuZWQgYnkKPiArwqDCoMKgwqBhIHByaW9y
IHN1Y2Nlc3NmdWwgY2FsbCB0byBWQVNfUlhfV0lOX09QRU4gaW9jdGwgKHNlZSBhYm92ZSkuIFRo
aXMKPiArwqDCoMKgwqBmaWVsZCBpcyB1c2VkIHRvIHBhaXIgdGhpcyBzZW5kIHdpbmRvdyB3aXRo
IGEgcmVjZWl2ZSB3aW5kb3cuIFRoZQo+ICvCoMKgwqDCoHByb2Nlc3MgbXVzdCBoYXZlIHN1ZmZp
Y2llbnQgcGVybWlzc2lvbnMgdG8gY29tbXVuaWNhdGUgd2l0aCB0aGUKPiArwqDCoMKgwqBwcm9j
ZXNzIG93bmluZyB0aGUgcmVjZWl2ZSB3aW5kb3cgaWRlbnRpZmllZCBieSByeF93aW5faGFuZGxl
LgoKQXMgYWJvdmUsIHRoaXMgc2hvdWxkIGJlIHBhcnQgb2YgdGhlIEZEIG90aGVyd2lzZSB1c2Vy
cyBjb3VsZCBzcGVjaWZ5IGFueXRoaW5nCmhlcmUgYW5kIHBhc3RlIHRvIGFueW9uZS4KCj4gK8Kg
wqDCoMKgVGhlIHRjX21vZGUgYW5kwqDCoHJzdmRfdHhidWYgZmllbGRzIGFyZSBjdXJyZW50bHkg
dW51c2VkIGFuZCBtdXN0IGJlCj4gK8KgwqDCoMKgc2V0IHRvIDAKPiArCj4gK8KgwqDCoMKgVGhl
IGZsYWdzIGZpZWxkIHNwZWNpZmllcyBhZGRpdGlvbmFsIGF0dHJpYnV0ZXMgdG8gdGhlIHdpbmRv
dy4gVGhlCj4gK8KgwqDCoMKgb25seSB2YWxpZCBiaXQgaW4gdGhlIGZsYWcgYXJlIGZvciBGVFcg
d2luZG93cyBpczoKPiArCj4gK8KgwqDCoMKgwqDCoMKgwqBWQVNfRkxBR1NfUElOX1dJTkRPV8Kg
wqDCoMKgaWYgc2V0LCBpbmRpY2F0ZXMgdGhlIGEgd2luZG93IHNob3VsZCBiZQo+ICvCoMKgwqDC
oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg
cGlubmVkIGluIGNhY2hlLiBUaGlzIGZsYWcgaXMgcmVzdHJpY3RlZAo+ICvCoMKgwqDCoMKgwqDC
oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgdG8gcHJp
dmlsZWdlZCB1c2Vycy4gU2VlIFBpbm5pbmcgd2luZG93cwo+ICvCoMKgwqDCoMKgwqDCoMKgwqDC
oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgYmVsb3cuCj4gKwo+
ICvCoMKgwqDCoEFsbCB0aGUgb3RoZXIgYml0cyBpbiB0aGUgZmxhZ3MgZmllbGQgbXVzdCBiZSBz
ZXQgdG8gMC4KPiArCj4gK8KgwqDCoMKgVGhlIGZpZWxkcyByZXNlcnZlZDEsIHJlc2VydmVkMiBh
bmQgcmVzZXJ2ZWQzIGFyZSBmb3IgZnV0dXJlIGV4dGVuc2lvbgo+ICvCoMKgwqDCoGFuZCBtdXN0
IGJlIHNldCB0byAwLgo+ICsKPiArwqDCoMKgwqBSZXR1cm4gdmFsdWU6Cj4gKwo+ICvCoMKgwqDC
oFRoZSBWQVNfVFhfV0lOX09QRU4gaW9jdGwgcmV0dXJucyAwIG9uIHN1Y2Nlc3MuIE9uIGVycm9y
LCBpdCByZXR1cm5zIC0xCj4gK8KgwqDCoMKgYW5kIHNldHMgdGhlIGVycm5vIHZhcmlhYmxlIHRv
IGluZGljYXRlIHRoZSBlcnJvci4KPiArCj4gK8KgwqDCoMKgRXJyb3IgY29uZGl0aW9uczoKPiAr
Cj4gK8KgwqDCoMKgwqDCoMKgwqBFSU5WQUzCoMKgwqDCoMKgwqB2ZXJzaW9uLCB2YXNfaWQgb3Ig
cnhfd2luX2hhbmRsZSBmaWVsZHMgYXJlIGludmFsaWQKPiArCj4gK8KgwqDCoMKgwqDCoMKgwqBF
SU5WQUzCoMKgwqDCoMKgwqBmZCBkb2VzIG5vdCByZWZlciB0byBhIHZhbGlkIFZBUyBkZXZpY2Uu
Cj4gKwo+ICvCoMKgwqDCoMKgwqDCoMKgRUlOVkFMwqDCoMKgwqDCoMKgZmQgaXMgYWxyZWFkeSBh
c3NvY2lhdGVkIHdpdGggYSByZWNlaXZlIHdpbmRvdwo+ICsKPiArwqDCoMKgwqDCoMKgwqDCoEVO
T1NQQ8KgwqDCoMKgwqDCoFN5c3RlbSBoYXMgdG9vIG1hbnkgYWN0aXZlIHdpbmRvd3MgKGNvbm5l
Y3Rpb25zKSBvcGVuLAo+ICsKPiArwqDCoMKgwqDCoMKgwqDCoEVJTlZBTMKgwqDCoMKgwqDCoEZv
ciBGVFcgd2luZG93cywgcnN2ZF90eGJ1ZiBpcyBub3QgMC4KPiArCj4gK8KgwqDCoMKgwqDCoMKg
wqBFSU5WQUzCoMKgwqDCoMKgwqBGb3IgRlRXIHdpbmRvd3MsIHRjX21vZGUgaXMgbm90IFZBU19U
SFJFU0hfRElTQUJMRUQuCj4gKwo+ICvCoMKgwqDCoMKgwqDCoMKgRVBFUk3CoMKgwqDCoMKgwqDC
oFZBU19GTEFHU19QSU5fV0lORE9XIGlzIHNldCBpbiAnZmxhZ3MnIGZpZWxkIGFuZCBwcm9jZXNz
Cj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqBpcyBub3QgcHJpdmls
ZWdlZC4KPiArCj4gK8KgwqDCoMKgwqDCoMKgwqBFUEVSTcKgwqDCoMKgwqDCoMKgVkFTX0ZMQUdT
X0hJR0hfUFJJIGlzIHNldCBpbiAnZmxhZ3MnIGZpZWxkIGFuZCBwcm9jZXNzCj4gK8KgwqDCoMKg
wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqBpcyBub3QgcHJpdmlsZWdlZC4KPiArCj4g
K8KgwqDCoMKgwqDCoMKgwqBFSU5WQUzCoMKgwqDCoMKgwqBhbiBpbnZhbGlkIGZsYWcgaXMgc2V0
IGluIHRoZSAnZmxhZ3MnIGZpZWxkLiAoRm9yIEZUVwo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoMKgwqDCoMKgwqDCoMKgd2luZG93cywgVkFTX0ZMQUdTX0hJR0hfUFJJIGlzIGFsc28gaW52
YWxpZCkuCj4gKwo+ICvCoMKgwqDCoMKgwqDCoMKgRUlOVkFMwqDCoMKgwqDCoMKgcmVzZXJ2ZWQg
ZmllbGRzIGFyZSBub3Qgc2V0IHRvIDAuCj4gKwo+ICvCoMKgwqDCoFNlZSB0aGUgaW9jdGwoMikg
bWFuIHBhZ2UgZm9yIG1vcmUgZGV0YWlscywgZXJyb3IgY29kZXMgYW5kIHJlc3RyaWN0aW9ucy4K
PiArCj4gKzQuIG1tYXAoKSBOWC1GVFcgZGV2aWNlIGZkCj4gKwo+ICvCoMKgwqDCoFRoZSBtbWFw
KCkgc3lzdGVtIGNhbGwgZm9yIGEgTlgtRlRXIGRldmljZSBmZCByZXR1cm5zIGEgInBhc3RlIGFk
ZHJlc3MiCj4gK8KgwqDCoMKgdGhhdCB0aGUgYXBwbGljYXRpb24gY2FuIHVzZSB0byBDT1BZL1BB
U1RFIGEgQ1JCIHRvIHRoZSB3YWl0aW5nIHRocmVhZC4KPiArCj4gK8KgwqDCoMKgwqDCoMKgwqBw
YXN0ZV9hZGRyID0gbW1hcChOVUxMLCBzaXplLCBwcm90LCBmbGFncywgZmQsIG9mZnNldCk7Cj4g
Kwo+ICvCoMKgwqDCoFRoZSBtbWFwKCkgb3BlcmF0aW9uIGlzIG9ubHkgdmFsaWQgb24gYSBmaWxl
IGRlc2NyaXB0b3IgYXNzb2NpYXRlZAo+ICvCoMKgwqDCoHdpdGggYSBzZW5kIHdpbmRvdy4KPiAr
Cj4gK8KgwqDCoMKgT25seSByZXN0cmljdGlvbnMgb24gbW1hcCBmb3IgYSBOWC1GVFcgZGV2aWNl
IGZkIGFyZToKPiArCj4gK8KgwqDCoMKgwqDCoMKgwqAtIHNpemUgcGFyYW1ldGVyIHNob3VsZCBi
ZSBvbmUgcGFnZSBzaXplCj4gKwo+ICvCoMKgwqDCoMKgwqDCoMKgLSBvZmZzZXQgcGFyYW1ldGVy
IHNob3VsZCBiZSAwVUxMLgo+ICsKPiArwqDCoMKgwqBSZWZlciB0byBtbWFwKDIpIG1hbiBwYWdl
IGZvciBhZGRpdGlvbmFsIGRldGFpbHMvcmVzdHJpY3Rpb25zLgo+ICsKPiArwqDCoMKgwqBJbiBh
ZGRpdGlvbiB0byB0aGUgZXJyb3IgY29uZGl0aW9ucyBsaXN0ZWQgb24gdGhlIG1tYXAoMikgbWFu
IHBhZ2UsCj4gK8KgwqDCoMKgbW1hcCgpIGNhbiBhbHNvIGZhaWwgd2l0aCBvbmUgb2YgZm9sbG93
aW5nIGVycm9yIGNvZGVzOgo+ICsKPiArwqDCoMKgwqDCoMKgwqDCoEVJTlZBTMKgwqDCoMKgwqDC
oGZkIGlzIG5vdCBhc3NvY2lhdGVkIHdpdGggYW4gb3BlbiBzZW5kIHdpbmRvdyAoaS5lIG1tYXAo
KQo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgZG9lcyBub3QgZm9s
bG93IGEgc3VjY2Vzc2Z1bCBjYWxsIHRvIHRoZSBWQVNfVFhfV0lOX09QRU4KPiArwqDCoMKgwqDC
oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoGlvY3RsKS4KPiArCj4gK8KgwqDCoMKgwqDC
oMKgwqBFSU5WQUzCoMKgwqDCoMKgwqBvZmZzZXQgZmllbGQgaXMgbm90IDBVTEwuCj4gKwo+ICsK
PiArNS4gVkFTIElECj4gKwo+ICvCoMKgwqDCoEEgc3lzdGVtIG1heSBoYXZlIHNldmVyYWwgaW5z
dGFuY2VzIG9mIFZBUyBpbiB0aGUgaGFyZHdhcmUsIHR5cGljYWxseQo+ICvCoMKgwqDCoG9uZSBw
ZXIgUE9XRVIgOSBjaGlwLiBUaGUgY2hvaWNlIG9mIGEgc3BlY2lmaWMgaW5zdGFuY2Ugb2YgVkFT
IGNhbiBoYXZlCj4gK8KgwqDCoMKgc2lnbmlmaWNhbnQgaW1wYWN0IG9uIHRoZSBwZXJmb3JtYW5j
ZSwgc3BlY2lhbGx5IGlmIHRoZSBhcHBsaWNhdGlvbgo+ICvCoMKgwqDCoG1pZ3JhdGVzIGZyb20g
b25lIENQVSB0byBhbm90aGVyLiBBcHBsaWNhdGlvbnMgY2FuIHNwZWNpZnkgYSB2YXNfaWQKPiAr
wqDCoMKgwqB1c2luZyB0aGUgVkFTX1RYX1dJTl9PUEVOIGFuZCBWQVNfUlhfV0lOX09QRU4gaW9j
dGxzIGFuZCBzaG91bGQgYmUKPiArwqDCoMKgwqBwcnVkZW50IGluIGNob29zaW5nIGFuIGluc3Rh
bmNlIG9mIFZBUy4KPiArCj4gK8KgwqDCoMKgVGhlIHZhc19pZCBmb3IgZWFjaCBpbnN0YW5jZSBv
ZiBWQVMgaXMgbGlzdGVkIGFzIHRoZSBkZXZpY2UgdHJlZQo+ICvCoMKgwqDCoHByb3BlcnR5ICdp
Ym0sdmFzLWlkJy4gRGV0ZXJtaW5pbmcgdGhlIHNwZWNpZmljIHZhc19pZCB0byB1c2UgZm9yCj4g
K8KgwqDCoMKgYSBzcGVjaWZpYyBhcHBsaWNhdGlvbiB0aHJlYWQgaXMgYmV5b25kIHRoZSBzY29w
ZSBvZiB0aGlzIEFQSS4KCkkgd291bGQgbGVhbiB0b3dhcmRzIGhhdmluZyAxIGRldmljZSBwZXIg
dmFzL2NoaXAgYnV0IEknbGwgZGVmZXIgdG8gbXBlIGFuZCBiZW5oCm9uIHRoZSBiZXN0IG9wdGlv
biBoZXJlLgoKeW91IHBsYW5uaW5nIGEgbGliZnR3IHRvIGRvIHRoaXM/Cgo+ICsKPiArwqDCoMKg
wqBJZiB0aGUgYXBwbGljYXRpb24gaGFzIG5vIHByZWZlcmVuY2UsIHRoZSB2YXNfaWQgZmllbGQg
bWF5IGJlIHNldCB0bwo+ICvCoMKgwqDCoC0xIGFuZCB0aGUga2VybmVsIHdpbGwgY2hvb3NlIGEg
c3VpdGFibGUgaW5zdGFuY2Ugb2YgdGhlIFZBUyBlbmdpbmUuCgorMSAKCj4gKzYuIENPUFkvUEFT
VEUgb3BlcmF0aW9uczoKPiArCj4gK8KgwqDCoMKgQXBwbGljYXRpb25zIHNob3VsZCB1c2UgdGhl
IENPUFkgYW5kIFBBU1RFIGluc3RydWN0aW9ucyBkZWZpbmVkIGluCj4gK8KgwqDCoMKgdGhlIFJG
QyB0byBjb3B5L3Bhc3RlIHRoZSBDUkIuIEZvciBWQVMvRlRXIHVzYWdlLCB0aGUgY29udGVudHMg
b2YKPiArwqDCoMKgwqBDUkIgaWYgYW55LCBhcmUgaWdub3JlZC4gQ1JCIGNhbiBiZSBOVUxMLgo+
ICsKPiArNy4gSW50ZXJydXB0IGNvbXBsZXRpb24gYW5kIHNpZ25hbCBoYW5kbGluZwo+ICsKPiAr
wqDCoMKgwqBObyBWQVMtc3BlY2lmaWMgc2lnbmFscyB3aWxsIGJlIGdlbmVyYXRlZCB0byB0aGUg
YXBwbGljYXRpb24gdGhyZWFkcwo+ICvCoMKgwqDCoHdpdGggdGhlIFZBUy9GVFcgdXNhZ2UuCgor
MQoKPiArCj4gKwo+ICs4LiBFeGFtcGxlL1Byb3Bvc2VkIHVzYWdlIG9mIHRoZSBWQVMvRlRXIEFQ
SQo+ICsKPiArwqDCoMKgwqBJbiB0aGUgZm9sbG93aW5nIGV4YW1wbGUgd2UgdXNlIHR3byB0aHJl
YWRzIHRoYXQgdXNlIHRoZSBWQVMvRlRXIEFQSS4KPiArwqDCoMKgwqBUaHJlYWQgVDEgdXNlcyB0
aGUgV0FJVCBpbnN0cnVjdGlvbiB0byB3YWl0IGZvciBhbiBldmVudC4gVGhyZWFkIFQyCj4gK8Kg
wqDCoMKgdXNlcyBjb3B5L3Bhc3RlIGluc3RydWN0aW9ucyB0byB3YWtlIHVwIFQxLgoKU28gaGVy
ZSdzIGhvdyBwc2V1ZG8gY29kZSBmb3IgbXkgaWRlYSB3b3VsZCBsb29rIHdpdGggcHRocmVhZHMu
IMKgCgpJJ3ZlIGFsc28gYWRkZWQgc29tZSBtZW1vcnkgYmFycmllcnMuIFRoZSBJU0Egc3VnZ2Vz
dHMgdGhhdCBjb3B5L3Bhc3RlIGhhcyBubwpvcmRlcmluZyBhc3NvY2lhdGVkIHdpdGggaXQsIHNv
IHlvdSBhcmUgZ29pbmcgdG8gbmVlZCB0aGVtIEkgdGhpbmsuIEknbSBub3Qgc3VyZQpvZiB0aGUg
Zmxhdm91ciB0aG91Z2guCgotLS0KYm9vbCBkb25lID0gZmFsc2U7CmludCByeGZkOwoKc3RhdGlj
IHZvaWQgcmVjaWV2ZXIodm9pZCkKewoJZG8gewrCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoGFzbSgid2FpdCIpOwoJCXNtcF9tYigpOyAvKiBuZWVkZWQgZm9yIHdhaXQgLT4gbWVtb3J5
wqDCoCovCgl9IHdoaWxlICghZG9uZSk7IC8qIGNoZWNrIGZvciBzcHVyaW91cyB3YWtldXAgKi8K
CS8qIHdva2VuIHVwISAqLwp9CgpzdGF0aWMgdm9pZCBzZW5kZXIodm9pZCkKewoJdm9pZCAqcGFz
dGVfYWRkcjsKCgkvKiBtbWFwIHRoZSByeCBmaWxlIGRlc2NyaXB0b3IgKi8KCXBhc3RlX2FkZHIg
PSBtbWFwKE5VTEwsIGdldHBhZ2VzaXplKCksIHByb3QsIE1BUF9TSEFSRUQsIHJ4ZmQsIDApOwoK
CWRvbmUgPSB0cnVlOwoJc21wX21iKCk7IC8qIG5lZWRlZCBmb3IgbWVtb3J5IC0+IHBhc3RlICov
CsKgwqDCoMKgwqDCoMKgwqB3cml0ZV9jcmIocGFzdGVfYWRkcik7Cn0KCmludCBtYWluKCkKewoJ
cHRocmVhZF90IHRocmVhZDsKCWludCBkZXZmZDsKCsKgwqDCoMKgwqDCoMKgwqBkZXZmZCA9IG9w
ZW4oIi9kZXYvdmFzLWZ0dyIsIE9fUkRXUik7CgoJLyogY3JlYXRlIGEgbmV3IHJ4IGZpbGUgZGVz
Y3JpcHRvciBhc3NvY2lhdGVkIHdpdGggdGhpcyBMUElEL1BJRC9USUQgKi8KwqDCoMKgwqDCoMKg
wqDCoHJ4ZmQgPSBpb2N0bChkZXZmZCwgVkFTX1JYX0NSRUFURSk7CgoJcHRocmVhZF9jcmVhdGUo
JnRocmVhZCwgTlVMTCwgc2VuZGVyLCBOVUxMKTsKCgkvKiBSZWNpZXZlciBtdXN0ICpub3QqIGJl
IGEgbmV3IHRocmVhZCBzaW5jZSBWQVNfUlhfQ1JFQVRFCgnCoMKgwqBpb2N0bCBpcyBhc3NvY2lh
dGVkIHdpdGggdGhpcyBMUElEL1BJRC9USUTCoAoJKi8KCXJlY2lldmVyKCk7Cn0KCl==
WARNING: multiple messages have this Message-ID (diff)
From: Michael Neuling <mikey@neuling.org>
To: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>,
Michael Ellerman <mpe@ellerman.id.au>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>,
stewart@linux.vnet.ibm.com, apopple@au1.ibm.com,
hbabu@us.ibm.com, oohall@gmail.com, linuxppc-dev@ozlabs.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v6 17/17] powerpc/vas: Document FTW API/usage
Date: Mon, 14 Aug 2017 20:43:30 +1000 [thread overview]
Message-ID: <1502707410.9844.10.camel@neuling.org> (raw)
In-Reply-To: <1502233622-9330-18-git-send-email-sukadev@linux.vnet.ibm.com>
On Tue, 2017-08-08 at 16:07 -0700, Sukadev Bhattiprolu wrote:
> Document the usage of the VAS Fast thread-wakeup API.
>
> Thanks for input/comments from Benjamin Herrenschmidt, Michael Neuling,
> Michael Ellerman, Robert Blackmore, Ian Munsie, Haren Myneni, Paul Mackerras.
>
> Cc:Ian Munsie <imunsie@au1.ibm.com>
> Cc:Paul Mackerras <paulus@ozlabs.org>
> Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> ---
> Documentation/powerpc/ftw-api.txt | 373
> ++++++++++++++++++++++++++++++++++++++
> 1 file changed, 373 insertions(+)
> create mode 100644 Documentation/powerpc/ftw-api.txt
>
> diff --git a/Documentation/powerpc/ftw-api.txt b/Documentation/powerpc/ftw-
> api.txt
> new file mode 100644
> index 0000000..0b3f16f
> --- /dev/null
> +++ b/Documentation/powerpc/ftw-api.txt
> @@ -0,0 +1,373 @@
> +Virtual Accelerator Switchboard and Fast Thread-Wakeup API
> +
> + Power9 processor supports a hardware subystem known as the Virtual
> + Accelerator Switchboard (VAS) which allows two entities in the Power9
> + system to efficiently exchange messages. Messages must be formatted as
> + Coprocessor Reqeust Blocks (CRB) and be submitted using the COPY/PASTE
> + instructions (new in Power9).
> +
> + Usage of VAS depends on the entities exchanging the messages and
> + currently two usages have been identified.
> +
> + First usage of VAS, referred to as VAS/NX involves a software thread
> + submitting data compression requests to a co-processor (hardware/nest
> + accelerator) aka NX engine. The API for this usage is described in the
> + VAS/NX API document.
> +
> + Alternatively, VAS can be used by two software threads to efficiently
> + exchange messages. Initially, this mechanism is intended to wake up a
> + waiting thread quickly - i.e "fast thread wake-up (FTW)". This document
> + describes the user API for this VAS/FTW mechanism.
> +
> + Application access to the FTW mechanism is provided through the NX-FTW
> + device node (/dev/crypto/nx-ftw) implemented by the VAS/FTW device
> + driver.
crypto?
> +
> + A software thread T1 that intends to wait for an event must first setup
> + a receive window, by opening the NX-FTW device and using the
> + VAS_RX_WIN_OPEN ioctl. Upon successful return from the VAS_RX_WIN_OPEN
> + ioctl, an rx_win_handle is returned.
I realise there is a window here as part of the hardware implementation, but the
users don't care about the window on the receive side. It's hidden from them.
It's just an rx handle IMHO.
The sender certainly has a window that users care about since they have to mmap
it.
> +
> + A software thread T2 that intends to wake up T1 at some point, must first
> + set up a "send window" using the VAS_TX_WIN_OPEN ioctl and specify the
> + rx_win_handle obtained by T1. After a successful VAS_TX_WIN_OPEN ioctl
> the
> + send window of T2 is considered paired with the receive window of T1. The
> + thread T2 must then use mmap() to obtain a "paste address" for the send
> + window.
> + With this set up, thread T1 can wait for an event using the WAIT
> + instruction.
> +
> + Thread T2 can wake up T1 by using the "COPY/PASTE" instructions and
> + submitting an empty/NULL CRB to the send window's paste address. The
> + wait/wake up process can be repeated as long as the threads have the
> + send/receive windows open.
> +1. NX-FTW Device Node
> +
> + There is one /dev/crypto/nx-ftw node in the system and it provides
> + access to the VAS/FTW functionality.
> + The only valid operations on the NX-FTW node are:
> +
> + - open() the device for read and write.
> +
> + - issue either VAS_RX_WIN_OPEN or VAS_TX_WIN_OPEN ioctls to set up
> + receive or send (only one of them per open).
> +
> + - if the open is associated with send window (i.e VAS_TX_WIN_OPEN
> + ioctl was issued) mmap() the send window into the application's
> + virtual address space. (i.e get a 'paste_address' for the send
> + window).
> +
> + - close the device node.
> +
> + Other file operations on the NX-FTW node are undefined.
> +
> + Note tHAT the COPY and PASTE operations go directly to the hardware
> + and not go through the NX-FTW device.
I don't understand this statement
> +
> + Although a system may have several instances of the VAS in the system
> + (typically, one per P9 chip) there is just one NX-FTW device node in
> + the system.
> + When the NX-FTW device node is opened, the kernel assigns a suitable
> + instance of VAS to the process. Kernel will make a best-effort
> attempt
> + to assign an optimal instance of VAS for the process. In the initial
> + release, the kernel does not support migrating the VAS instance if the
> + process migrates from a processor on one chip to a processor on another
> + chip.
How is it "optimal"?
> + Applications may chose a specific instance of the VAS using the 'vas_id'
> + field in the VAS_TX_WIN_OPEN and VAS_RX_WIN_OPEN ioctls as detailed
> below.
> +2. Open NX-FTW node
> +
> + The device should be opened for read and write. No special privileges
> + are needed to open the device. The device may be opened multiple times.
> +
> + Each open() of the NX-FTW device may be associated with either a send
> + window or receive window but not both.
> +
> + See open(2) system call man pages for other details such as return
> + values, error codes and restrictions.
> +
> +3. Setup Receive window (VAS_RX_WIN_OPEN ioctl)
> +
> + A thread that expects to wait for events and be woken up using COPY/PASTE
> + must first set up a receive window by issuing the VAS_RX_WIN_OPEN ioctl.
> +
> + #include <asm/vas.h>
> +
> + struct vas_rx_win_open_attr rxattr;
> +
> + rc = ioctl(fd, VAS_RX_WIN_OPEN, &rxattr);
> +
> + The attributes of rxattr are as follows:
> +
> + struct vas_rx_win_open_attr {
> + int16_t version;
> + int16_t vas_id;
> + int32_t rx_win_handle; /* output field */
> + int64_t reserved[8];
> + };
> +
> + The version field identifies the version of the API and must currently
> + be set to 1.
> +
> + The vas_id field identifies a specific instance of the VAS that the
> + application wishes to access. See section on VAS ID below.
> +
> + The reserved field must be set to all zeroes.
> +
> + Upon successful return from the ioctl, the rx_win_handle field contains
> + an identifier for the VAS window associated with this "sleeping" thread.
> +
> + This rx_win_handle field is used to "pair" this receive window with a
> + send window and must be specified when opening the corresponding send
> + window (see struct vas_tx_win_open_attr below).
> +
> + Return value:
> +
> + The VAS_RX_WIN_OPEN ioctl returns 0 on success. On error, it returns -1
> + and sets the errno variable to indicate the error.
> +
> + Error codes:
> +
> + EINVAL version is invalid
> +
> + EINVAL vas_id is invalid
> +
> + EINVAL reserved field is not set to zeroes
> +
> + EINVAL fd is already associated with a send window
> +
> +
> +3. Set up a Send window (VAS_TX_WIN_OPEN ioctl)
> +
> + An application thread that expects to wake up a waiting thread using
> + copy/paste, must first set up a send window that is paired with the
> + receive window of the waiting thread. This is accomplished using the
> + VAS_TX_WIN_OPEN ioctl.
> +
> + #include <asm/vas.h>
> +
> + struct vas_tx_win_open_attr txattr;
> +
> + rc = ioctl(fd, VAS_TX_WIN_OPEN, &txattr);
So we talked about this offline before.... the fd here should not be from the
/dev device but should be the fd from rx_win_open ioctl.
As you have it here you pass the handle in as a parameter of ioctl. This means
all the permissions checks have to be done by you as to if these two windows can
be linked. If you use the fd from before, you can assume if the receiver has
given this fd to the sender, it has the right permissions.
I have some pseudo code at the end shows this.
> + The attributes 'txattr' for the VAS_TX_WIN_OPEN ioctl are defined as
> + follows:
> +
> + struct vas_tx_win_open_attr {
> + int32_t version;
> + int16_t vas_id;
> + uint32_t rx_win_handle;
> +
> + int64_t reserved1;
> +
> + int64_t flags;
> + int64_t reserved2;
> +
> + int32_t tc_mode;
> + int32_t rsvd_txbuf;
> + int64_t reserved3[6];
> + };
> +
> + The version field must currently be set to 1.
> +
> + The vas_id field identifies a specific instance of the VAS that the
> + application wishes to access. See section on VAS ID below.
Can this be different to the rx?
> + The rx_win_handle field must be set to the rx_win_handle returned by
> + a prior successful call to VAS_RX_WIN_OPEN ioctl (see above). This
> + field is used to pair this send window with a receive window. The
> + process must have sufficient permissions to communicate with the
> + process owning the receive window identified by rx_win_handle.
As above, this should be part of the FD otherwise users could specify anything
here and paste to anyone.
> + The tc_mode and rsvd_txbuf fields are currently unused and must be
> + set to 0
> +
> + The flags field specifies additional attributes to the window. The
> + only valid bit in the flag are for FTW windows is:
> +
> + VAS_FLAGS_PIN_WINDOW if set, indicates the a window should be
> + pinned in cache. This flag is restricted
> + to privileged users. See Pinning windows
> + below.
> +
> + All the other bits in the flags field must be set to 0.
> +
> + The fields reserved1, reserved2 and reserved3 are for future extension
> + and must be set to 0.
> +
> + Return value:
> +
> + The VAS_TX_WIN_OPEN ioctl returns 0 on success. On error, it returns -1
> + and sets the errno variable to indicate the error.
> +
> + Error conditions:
> +
> + EINVAL version, vas_id or rx_win_handle fields are invalid
> +
> + EINVAL fd does not refer to a valid VAS device.
> +
> + EINVAL fd is already associated with a receive window
> +
> + ENOSPC System has too many active windows (connections) open,
> +
> + EINVAL For FTW windows, rsvd_txbuf is not 0.
> +
> + EINVAL For FTW windows, tc_mode is not VAS_THRESH_DISABLED.
> +
> + EPERM VAS_FLAGS_PIN_WINDOW is set in 'flags' field and process
> + is not privileged.
> +
> + EPERM VAS_FLAGS_HIGH_PRI is set in 'flags' field and process
> + is not privileged.
> +
> + EINVAL an invalid flag is set in the 'flags' field. (For FTW
> + windows, VAS_FLAGS_HIGH_PRI is also invalid).
> +
> + EINVAL reserved fields are not set to 0.
> +
> + See the ioctl(2) man page for more details, error codes and restrictions.
> +
> +4. mmap() NX-FTW device fd
> +
> + The mmap() system call for a NX-FTW device fd returns a "paste address"
> + that the application can use to COPY/PASTE a CRB to the waiting thread.
> +
> + paste_addr = mmap(NULL, size, prot, flags, fd, offset);
> +
> + The mmap() operation is only valid on a file descriptor associated
> + with a send window.
> +
> + Only restrictions on mmap for a NX-FTW device fd are:
> +
> + - size parameter should be one page size
> +
> + - offset parameter should be 0ULL.
> +
> + Refer to mmap(2) man page for additional details/restrictions.
> +
> + In addition to the error conditions listed on the mmap(2) man page,
> + mmap() can also fail with one of following error codes:
> +
> + EINVAL fd is not associated with an open send window (i.e mmap()
> + does not follow a successful call to the VAS_TX_WIN_OPEN
> + ioctl).
> +
> + EINVAL offset field is not 0ULL.
> +
> +
> +5. VAS ID
> +
> + A system may have several instances of VAS in the hardware, typically
> + one per POWER 9 chip. The choice of a specific instance of VAS can have
> + significant impact on the performance, specially if the application
> + migrates from one CPU to another. Applications can specify a vas_id
> + using the VAS_TX_WIN_OPEN and VAS_RX_WIN_OPEN ioctls and should be
> + prudent in choosing an instance of VAS.
> +
> + The vas_id for each instance of VAS is listed as the device tree
> + property 'ibm,vas-id'. Determining the specific vas_id to use for
> + a specific application thread is beyond the scope of this API.
I would lean towards having 1 device per vas/chip but I'll defer to mpe and benh
on the best option here.
you planning a libftw to do this?
> +
> + If the application has no preference, the vas_id field may be set to
> + -1 and the kernel will choose a suitable instance of the VAS engine.
+1
> +6. COPY/PASTE operations:
> +
> + Applications should use the COPY and PASTE instructions defined in
> + the RFC to copy/paste the CRB. For VAS/FTW usage, the contents of
> + CRB if any, are ignored. CRB can be NULL.
> +
> +7. Interrupt completion and signal handling
> +
> + No VAS-specific signals will be generated to the application threads
> + with the VAS/FTW usage.
+1
> +
> +
> +8. Example/Proposed usage of the VAS/FTW API
> +
> + In the following example we use two threads that use the VAS/FTW API.
> + Thread T1 uses the WAIT instruction to wait for an event. Thread T2
> + uses copy/paste instructions to wake up T1.
So here's how pseudo code for my idea would look with pthreads.
I've also added some memory barriers. The ISA suggests that copy/paste has no
ordering associated with it, so you are going to need them I think. I'm not sure
of the flavour though.
---
bool done = false;
int rxfd;
static void reciever(void)
{
do {
asm("wait");
smp_mb(); /* needed for wait -> memory */
} while (!done); /* check for spurious wakeup */
/* woken up! */
}
static void sender(void)
{
void *paste_addr;
/* mmap the rx file descriptor */
paste_addr = mmap(NULL, getpagesize(), prot, MAP_SHARED, rxfd, 0);
done = true;
smp_mb(); /* needed for memory -> paste */
write_crb(paste_addr);
}
int main()
{
pthread_t thread;
int devfd;
devfd = open("/dev/vas-ftw", O_RDWR);
/* create a new rx file descriptor associated with this LPID/PID/TID */
rxfd = ioctl(devfd, VAS_RX_CREATE);
pthread_create(&thread, NULL, sender, NULL);
/* Reciever must *not* be a new thread since VAS_RX_CREATE
ioctl is associated with this LPID/PID/TID
*/
reciever();
}
next prev parent reply other threads:[~2017-08-14 10:43 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-08-08 23:06 [PATCH v6 00/17] powerpc/vas: Enable VAS Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 01/17] powerpc/vas: Define macros, register fields and structures Sukadev Bhattiprolu
2017-08-14 5:21 ` Michael Ellerman
2017-08-14 6:05 ` Nicholas Piggin
2017-08-14 11:00 ` Michael Ellerman
2017-08-14 19:14 ` Sukadev Bhattiprolu
2017-08-16 12:07 ` Michael Ellerman
2017-08-16 23:07 ` Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 02/17] powerpc/vas: Move GET_FIELD/SET_FIELD to vas.h Sukadev Bhattiprolu
2017-08-14 7:26 ` Michael Ellerman
2017-08-08 23:06 ` [PATCH v6 03/17] powerpc/vas: Define vas_init() and vas_exit() Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 04/17] powerpc/vas: Define helpers to access MMIO regions Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 05/17] powerpc/vas: Define helpers to init window context Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 06/17] powerpc/vas: Define helpers to alloc/free windows Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 07/17] powerpc/vas: Define vas_win_paste_addr() Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 08/17] powerpc/vas: Define vas_win_id() Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 09/17] powerpc/vas: Define vas_rx_win_open() interface Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 10/17] " Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 11/17] powerpc/vas: Define vas_win_close() interface Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 12/17] powerpc/vas: Define vas_tx_win_open() Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 13/17] powerpc/vas: Define copy/paste interfaces Sukadev Bhattiprolu
2017-08-08 23:06 ` [PATCH v6 14/17] powerpc: Add support for setting SPRN_TIDR Sukadev Bhattiprolu
2017-08-14 7:02 ` Michael Neuling
2017-08-14 7:02 ` Michael Neuling
2017-08-14 13:30 ` Benjamin Herrenschmidt
2017-08-14 19:27 ` Sukadev Bhattiprolu
2017-08-14 11:16 ` Michael Ellerman
2017-08-14 20:03 ` Sukadev Bhattiprolu
2017-08-14 22:25 ` Benjamin Herrenschmidt
2017-08-14 22:23 ` Benjamin Herrenschmidt
2017-08-08 23:07 ` [PATCH v6 15/17] powerpc/vas: Define window open ioctls API Sukadev Bhattiprolu
2017-08-08 23:07 ` [PATCH v6 16/17] powerpc/vas: Implement a simple FTW driver Sukadev Bhattiprolu
2017-08-14 6:53 ` Michael Ellerman
2017-08-08 23:07 ` [PATCH v6 17/17] powerpc/vas: Document FTW API/usage Sukadev Bhattiprolu
2017-08-14 10:43 ` Michael Neuling [this message]
2017-08-14 10:43 ` Michael Neuling
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1502707410.9844.10.camel@neuling.org \
--to=mikey@neuling.org \
--cc=apopple@au1.ibm.com \
--cc=benh@kernel.crashing.org \
--cc=hbabu@us.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxppc-dev@ozlabs.org \
--cc=mpe@ellerman.id.au \
--cc=oohall@gmail.com \
--cc=stewart@linux.vnet.ibm.com \
--cc=sukadev@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.