From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Martin Subject: [PATCH v4 11/28] arm64/sve: Core task context handling Date: Fri, 27 Oct 2017 11:50:53 +0100 Message-ID: <1509101470-7881-12-git-send-email-Dave.Martin@arm.com> References: <1509101470-7881-1-git-send-email-Dave.Martin@arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <1509101470-7881-1-git-send-email-Dave.Martin@arm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu To: linux-arm-kernel@lists.infradead.org Cc: linux-arch@vger.kernel.org, Okamoto Takayuki , libc-alpha@sourceware.org, Ard Biesheuvel , Szabolcs Nagy , Catalin Marinas , Will Deacon , kvmarm@lists.cs.columbia.edu List-Id: linux-arch.vger.kernel.org VGhpcyBwYXRjaCBhZGRzIHRoZSBjb3JlIHN1cHBvcnQgZm9yIHN3aXRjaGluZyBhbmQgbWFuYWdp bmcgdGhlIFNWRQphcmNoaXRlY3R1cmFsIHN0YXRlIG9mIHVzZXIgdGFza3MuCgpDYWxscyB0byB0 aGUgZXhpc3RpbmcgRlBTSU1EIGxvdy1sZXZlbCBzYXZlL3Jlc3RvcmUgZnVuY3Rpb25zIGFyZQpm YWN0b3JlZCBvdXQgYXMgbmV3IGZ1bmN0aW9ucyB0YXNrX2Zwc2ltZF97c2F2ZSxsb2FkfSgpLCBz aW5jZSBTVkUKbm93IGR5bmFtaWNhbGx5IG1heSBvciBtYXkgbm90IG5lZWQgdG8gYmUgaGFuZGxl ZCBhdCB0aGVzZSBwb2ludHMKZGVwZW5kaW5nIG9uIHRoZSBrZXJuZWwgY29uZmlndXJhdGlvbiwg aGFyZHdhcmUgZmVhdHVyZXMgZGlzY292ZXJlZAphdCBib290LCBhbmQgdGhlIHJ1bnRpbWUgc3Rh dGUgb2YgdGhlIHRhc2suICBUbyBtYWtlIHRoZXNlCmRlY2lzaW9ucyBhcyBmYXN0IGFzIHBvc3Np YmxlLCBjb25zdCBjcHVjYXBzIGFyZSB1c2VkIHdoZXJlCmZlYXNpYmxlLCB2aWEgdGhlIHN5c3Rl bV9zdXBwb3J0c19zdmUoKSBoZWxwZXIuCgpUaGUgU1ZFIHJlZ2lzdGVycyBhcmUgb25seSB0cmFj a2VkIGZvciB0aHJlYWRzIHRoYXQgaGF2ZSBleHBsaWNpdGx5CnVzZWQgU1ZFLCBpbmRpY2F0ZWQg YnkgdGhlIG5ldyB0aHJlYWQgZmxhZyBUSUZfU1ZFLiAgT3RoZXJ3aXNlLCB0aGUKRlBTSU1EIHZp ZXcgb2YgdGhlIGFyY2hpdGVjdHVyYWwgc3RhdGUgaXMgc3RvcmVkIGluCnRocmVhZC5mcHNpbWRf c3RhdGUgYXMgdXN1YWwuCgpXaGVuIGluIHVzZSwgdGhlIFNWRSByZWdpc3RlcnMgYXJlIG5vdCBz dG9yZWQgZGlyZWN0bHkgaW4KdGhyZWFkX3N0cnVjdCBkdWUgdG8gdGhlaXIgcG90ZW50aWFsbHkg bGFyZ2UgYW5kIHZhcmlhYmxlIHNpemUuCkJlY2F1c2UgdGhlIHRhc2tfc3RydWN0IHNsYWIgYWxs b2NhdG9yIG11c3QgYmUgY29uZmlndXJlZCB2ZXJ5CmVhcmx5IGR1cmluZyBrZXJuZWwgYm9vdCwg aXQgaXMgYWxzbyB0cmlja3kgdG8gY29uZmlndXJlIGl0CmNvcnJlY3RseSB0byBtYXRjaCB0aGUg bWF4aW11bSB2ZWN0b3IgbGVuZ3RoIHByb3ZpZGVkIGJ5IHRoZQpoYXJkd2FyZSwgc2luY2UgdGhp cyBkZXBlbmRzIG9uIGV4YW1pbmluZyBzZWNvbmRhcnkgQ1BVcyBhcyB3ZWxsIGFzCnRoZSBwcmlt YXJ5LiAgSW5zdGVhZCwgYSBwb2ludGVyIHN2ZV9zdGF0ZSBpbiB0aHJlYWRfc3RydWN0IHBvaW50 cwp0byBhIGR5bmFtaWNhbGx5IGFsbG9jYXRlZCBidWZmZXIgY29udGFpbmluZyB0aGUgU1ZFIHJl Z2lzdGVyIGRhdGEsCmFuZCBjb2RlIGlzIGFkZGVkIHRvIGFsbG9jYXRlIGFuZCBmcmVlIHRoaXMg YnVmZmVyIGF0IGFwcHJvcHJpYXRlCnRpbWVzLgoKVElGX1NWRSBpcyBzZXQgd2hlbiB0YWtpbmcg YW4gU1ZFIGFjY2VzcyB0cmFwIGZyb20gdXNlcnNwYWNlLCBpZgpzdWl0YWJsZSBoYXJkd2FyZSBz dXBwb3J0IGhhcyBiZWVuIGRldGVjdGVkLiAgVGhpcyBlbmFibGVzIFNWRSBmb3IKdGhlIHRocmVh ZDogYSBzdWJzZXF1ZW50IHJldHVybiB0byB1c2Vyc3BhY2Ugd2lsbCBkaXNhYmxlIHRoZSB0cmFw CmFjY29yZGluZ2x5LiAgSWYgc3VjaCBhIHRyYXAgaXMgdGFrZW4gd2l0aG91dCBzdWZmaWNpZW50 IHN5c3RlbS0Kd2lkZSBoYXJkd2FyZSBzdXBwb3J0LCBTSUdJTEwgaXMgc2VudCB0byB0aGUgdGhy ZWFkIGluc3RlYWQgYXMgaWYKYW4gdW5kZWZpbmVkIGluc3RydWN0aW9uIGhhZCBiZWVuIGV4ZWN1 dGVkOiB0aGlzIG1heSBoYXBwZW4gaWYKdXNlcnNwYWNlIHRyaWVzIHRvIHVzZSBTVkUgaW4gYSBz eXN0ZW0gd2hlcmUgbm90IGFsbCBDUFVzIHN1cHBvcnQKaXQgZm9yIGV4YW1wbGUuCgpUaGUga2Vy bmVsIHdpbGwgY2xlYXIgVElGX1NWRSBhbmQgZGlzYWJsZSBTVkUgZm9yIHRoZSB0aHJlYWQKd2hl bmV2ZXIgYW4gZXhwbGljaXQgc3lzY2FsbCBpcyBtYWRlIGJ5IHVzZXJzcGFjZS4gIEZvciBiYWNr d2FyZHMKY29tcGF0aWJpbGl0eSByZWFzb25zIGFuZCBjb25mb3JtYW5jZSB3aXRoIHRoZSBzcGly aXQgb2YgdGhlIGJhc2UKQUFyY2g2NCBwcm9jZWR1cmUgY2FsbCBzdGFuZGFyZCwgdGhlIHN1YnNl dCBvZiB0aGUgU1ZFIHJlZ2lzdGVyCnN0YXRlIHRoYXQgYWxpYXNlcyB0aGUgRlBTSU1EIHJlZ2lz dGVycyBpcyBzdGlsbCBwcmVzZXJ2ZWQgYWNyb3NzIGEKc3lzY2FsbCBldmVuIGlmIHRoaXMgaGFw cGVucy4gIFRoZSByZW1haW5kZXIgb2YgdGhlIFNWRSByZWdpc3RlcgpzdGF0ZSBsb2dpY2FsbHkg YmVjb21lcyB6ZXJvIGF0IHN5c2NhbGwgZW50cnksIHRob3VnaCB0aGUgYWN0dWFsCnplcm9pbmcg d29yayBpcyBjdXJyZW50bHkgZGVmZXJyZWQgdW50aWwgdGhlIHRocmVhZCBuZXh0IHRyaWVzIHRv CnVzZSBTVkUsIGNhdXNpbmcgYW5vdGhlciB0cmFwIHRvIHRoZSBrZXJuZWwuICBUaGlzIGltcGxl bWVudGF0aW9uCmlzIHN1Ym9wdGltYWw6IGluIHRoZSBmdXR1cmUsIHRoZSBmYXN0cGF0aCBjYXNl IG1heSBiZSBvcHRpbWlzZWQKdG8gemVybyB0aGUgcmVnaXN0ZXJzIGluLXBsYWNlIGFuZCBsZWF2 ZSBTVkUgZW5hYmxlZCBmb3IgdGhlIHRhc2ssCndoZXJlIGJlbmVmaWNpYWwuCgpUSUZfU1ZFIGlz IGFsc28gY2xlYXJlZCBpbiB0aGUgZm9sbG93aW5nIHNsb3dwYXRoIGNhc2VzLCB3aGljaCBhcmUK dGFrZW4gYXMgcmVhc29uYWJsZSBoaW50cyB0aGF0IHRoZSB0YXNrIG1heSBubyBsb25nZXIgdXNl IFNWRToKICogZXhlYwogKiBmb3JrIGFuZCBjbG9uZQoKQ29kZSBpcyBhZGRlZCB0byBzeW5jIGRh dGEgYmV0d2VlbiB0aHJlYWQuZnBzaW1kX3N0YXRlIGFuZAp0aHJlYWQuc3ZlX3N0YXRlIHdoZW5l dmVyIGVuYWJsaW5nL2Rpc2FibGluZyBTVkUsIGluIGEgbWFubmVyCmNvbnNpc3RlbnQgd2l0aCB0 aGUgU1ZFIGFyY2hpdGVjdHVyYWwgcHJvZ3JhbW1lcidzIG1vZGVsLgoKU2lnbmVkLW9mZi1ieTog RGF2ZSBNYXJ0aW4gPERhdmUuTWFydGluQGFybS5jb20+CkNjOiBBcmQgQmllc2hldXZlbCA8YXJk LmJpZXNoZXV2ZWxAbGluYXJvLm9yZz4KQ2M6IEFsZXggQmVubsOpZSA8YWxleC5iZW5uZWVAbGlu YXJvLm9yZz4KCi0tLQoKQ2hhbmdlcyBzaW5jZSB2MwotLS0tLS0tLS0tLS0tLS0tCgpDaGFuZ2Vz IHJlcXVlc3RlZCBieSBDYXRhbGluIE1hcmluYXM6CgogKiBDaGFuZ2Ugc3RhdGljIF9fbWF5YmVf dW51c2VkIGZ1bmN0aW9ucyB0byBzdGF0aWMgaW5saW5lcy4KCiAqIE1vdmUgZnBzaW1kX3JlbGVh c2VfdGhyZWFkKCkgdG8gdGhlIGFyY2hfcmVsZWFzZV90YXNrX3N0cnVjdCgpIHBhdGguCgogICBU aGlzIGF2b2lkcyBhbnkgbmVlZCB0byB3b3JyeSBhYm91dCBwcmVlbXB0aW9uLgoKICAgQSBkZWNs YXJhdGlvbiBmb3IgYXJjaF9yZWxlYXNlX3Rhc2tfc3RydWN0KCkgaXMgYWRkZWQgdG8KICAgPGFz bS90aHJlYWRfaW5mby5oPiB0byBoZWxwIGd1YXJkIGFnYWluc3QgaW5jb25zaXN0ZW50IHByb3Rv dHlwZXMsCiAgIHRob3VnaCB0aGlzIGlzIG5vdCByZXF1aXJlZCBhbmQgdGhlIGNvZGUgd291bGQg d29yayB3aXRob3V0IGl0LgoKICAgKHRocmVhZF9pbmZvLmggaXMgYWxyZWFkeSBpbmNsdWRlZCBi eSB0aGUgcmVsZXZhbnQgY29kZSBjb2RlLCBhbmQKICAgYXJjaC9zaCB1c2VzIHRoZSBzYW1lIGhl YWRlcikuCgogKiBBZGRlZCBjb21tZW50cyB0byBleHBsYWluIHRoZSBtZWFuaW5nIG9mIFRJRl9T VkUuCgogKiBBQkkgY2hhbmdlOiBaZXJvIFNWRSByZWdzIG9uIHN5c2NhbGwgZW50cnkuCgogICBE ZXRhaWxlZCByYXRpb25hbGU6CgpBcyBjdXJyZW50bHkgZG9jdW1lbnRlZCwgbm8gZ3VhcmFudGVl IGlzIG1hZGUgYWJvdXQgd2hhdCBhIHVzZXIKdGFzayBzZWVzIGluIHRoZSBTVkUgcmVnaXN0ZXJz IGFmdGVyIGEgc3lzY2FsbCwgZXhjZXB0IHRoYXQgVjAtVjMxCihjb3JyZXNwb25kaW5nIHRvIFow LVozMSBiaXRzIFsxMjc6MF0pIGFyZSBwcmVzZXJ2ZWQuCgpUaGUgYWN0dWFsIGtlcm5lbCBiZWhh dmlvdXIgY3VycmVudGx5IGltcGxlbWVudGVkIGlzIHRoYXQgdGhlIFNWRQpyZWdpc3RlcnMgYXJl IHplcm9lZCBpZiBhIGNvbnRleHQgc3dpdGNoIG9yIHNpZ25hbCBkZWxpdmVyeSBvY2N1cnMKZHVy aW5nIGEgc3lzY2FsbC4gIEFmdGVyIGEgZm9yaygpIG9yIGNsb25lKCksIHRoZSBTVkUgcmVnaXN0 ZXJzCm9mIHRoZSBjaGlsZCB0YXNrIGFyZSB6ZXJvZWQgYWxzby4gIFRoZSBTVkUgcmVnaXN0ZXJz IGFyZSBvdGhlcndpc2UKcHJlc2VydmVkLiAgRmxleGliaWxpdHkgaXMgcmV0YWluZWQgaW4gdGhl IEFCSSBhYm91dCB0aGUgZXhhY3QKY3JpdGVyaWEgZm9yIHRoZSBkZWNpc2lvbi4KClRoZXJlIGFy ZSBzb21lIHBvdGVudGlhbCBwcm9ibGVtcyB3aXRoIHRoaXMgYXBwcm9hY2guCgpTeXNjYWxsIGlt cGFjdAotLS0tLS0tLS0tLS0tLQoKV2lsbCwgQ2F0YWxpbiBhbmQgTWFyayBoYXZlIGV4cHJlc3Nl ZCBjb25jZXJucyBhYm91dCB0aGUgcmlzayBvZgpjcmVhdGluZyBkZSBmYWN0byBBQkkgaGVyZTog aW4gc2NlbmFyaW9zIG9yIHdvcmtsb2FkcyB3aGVyZSBhCmNvbnRleHQgc3dpdGNoIG5ldmVyIG9j Y3VycyBvciBpcyB2ZXJ5IHVubGlrZWx5LCB1c2Vyc3BhY2UgbWF5CmxlYXJuIHRvIHJlbHkgb24g cHJlc2VydmF0aW9uIG9mIHRoZSBTVkUgcmVnaXN0ZXJzIGFjcm9zcyBjZXJ0YWluCnN5c2NhbGxz LgoKSXQgaXMgZGlmZmljdWx0IHRvIGFzc2VzcyB0aGUgaW1wYWN0IG9mIHRoaXM6IHRoZSBzeXNj YWxsIEFCSSBpcwpub3QgYSBnZW5lcmFsLXB1cnBvc2UgaW50ZXJmYWNlLCBzaW5jZSBpdCBpcyB1 c3VhbGx5IGhpZGRlbiBiZWhpbmQKbGliYyB3cmFwcGVyczogZGlyZWN0IGludm9jYXRpb24gb2Yg U1ZDIGlzIGRpc2NvdXJhZ2VkLiAgSG93ZXZlciwKc3BlY2lhbGlzZWQgcnVudGltZXMsIHN0YXRp Y2FsbHkgbGlua2VkIHByb2dyYW1zIGFuZCBiaW5hcnkgYmxvYnMKbWF5IGJha2UgaW4gZGlyZWN0 IHN5c2NhbGxzIHRoYXQgbWFrZSBiYWQgYXNzdW1wdGlvbnMuCgpDb252ZXJzZWx5LCB0aGUgcmVs YXRpdmUgY29zdCBvZiB6ZXJvaW5nIHRoZSBTVkUgcmVncyB0byBtaXRpZ2F0ZQphZ2FpbnN0IHRo aXMgYWxzbyBjYW5ub3QgYmUgd2VsbCBjaGFyYWN0ZXJpc2VkIHVudGlsIFNWRSBoYXJkd2FyZQpl eGlzdHMuCgpwdHJhY2UgaW1wYWN0Ci0tLS0tLS0tLS0tLS0KClRoZSBjdXJyZW50IGltcGxlbWVu dGF0aW9uIGNhbiBkaXNjYXJkIGFuZCB6ZXJvIHRoZSBTVkUgcmVnaXN0ZXJzCmF0IGFueSBwb2lu dCBkdXJpbmcgYSBzeXNjYWxsLCBpbmNsdWRpbmcgYmVmb3JlLCBhZnRlciBvciBiZXR3ZWVuCnB0 cmFjZSB0cmFwcyBpbnNpZGUgYSBzaW5nbGUgc3lzY2FsbC4gIFRoaXMgbWVhbnMgdGhhdCBzZXR0 aW5nIHRoZQpTVkUgcmVnaXN0ZXJzIHRocm91Z2ggUFRSQUNFX1NFVFJFR1NFVCB3aWxsIG9mdGVu IG5vdCBkbyB3aGF0IHRoZQp1c2VyIGV4cGVjdHM6IHRoZSBuZXcgcmVnaXN0ZXIgdmFsdWVzIGFy ZSBvbmx5IGd1YXJhbnRlZWQgdG8Kc3Vydml2ZSBhcyBmYXIgYXMgdXNlcnNwYWNlIGlmIHNldCBm cm9tIGFuIGFzeW5jaHJvbm91cwpzaWduYWwtZGVsaXZlcnktc3RvcCAoZS5nLiwgYnJlYWtwb2lu dCwgU0VHViBvciBhc3luY2hyb25vdXMgc2lnbmFsCmRlbGl2ZXJlZCBvdXRzaWRlIHN5c2NhbGwg Y29udGV4dCkuCgpUaGlzIGlzIGNvbnNpc3RlbnQgd2l0aCB0aGUgY3VycmVudGx5IGRvY3VtZW50 ZWQgU1ZFIHVzZXIgQUJJLCBidXQKbGlrZWx5IHRvIGJlIHN1cnByaXNpbmcgZm9yIGEgZGVidWdn ZXIgdXNlciwgc2luY2Ugc2V0dGluZyBtb3N0CnJlZ2lzdGVycyBvZiBhIHRyYWNlZSBkb2Vzbid0 IGJlaGF2ZSBpbiB0aGlzIHdheS4KClRoaXMgcGF0Y2gKLS0tLS0tLS0tLQoKVGhlIGNvbW1vbiBz eXNjYWxsIGVudHJ5IHBhdGggaXMgbW9kaWZpZWQgdG8gZm9yY2libHkgZGlzY2FyZCBTVkUsCmFu ZCB0aGUgZGlzY2FyZCBsb2dpYyBlbHNld2hlcmUgaXMgcmVtb3ZlZC4KClRoaXMgbWVhbnMgdGhh dCB0aGVyZSBpcyBhIG1hbmRhdG9yeSBhZGRpdGlvbmFsIHRyYXAgdG8gdGhlIGtlcm5lbAp3aGVu IGEgdXNlciB0YXNrIHRyaWVzIHRvIHVzZSBTVkUgYWdhaW4gYWZ0ZXIgYSBzeXNjYWxsLiAgVGhp cyBjYW4KYmUgZXhwZW5zaXZlIGZvciBwcm9ncmFtcyB0aGF0IHVzZSBTVkUgaGVhdmlseSBhcm91 bmQgc3lzY2FsbHMsIGJ1dApjYW4gYmUgb3B0aW1pc2VkIGxhdGVyLgoKQmVjYXVzZSBhbGwgcHRy YWNlIHRyYXBzIG9jY3VyIGFmdGVyIHN5c2NhbGwgZW50cnksIHRoaXMgYWxzbyBtZWFucwp0aGF0 IG5vIGRpc2NhcmQgd2lsbCBvY2N1ciBhZnRlciBTVkUgcmVnaXN0ZXJzIGFyZSBzZXQgdGhyb3Vn aApwdHJhY2UsIHVudGlsIHRoZSB0YXNrIG5leHQgcGFzc2VzIHRocm91Z2ggdXNlcnNwYWNlIGFu ZCBkb2VzCmFub3RoZXIgc3lzY2FsbC4gIFRoZSBvbmx5IGV4Y2VwdGlvbiB0byB0aGlzIGlzIGlu IGZvcmsoKS9jbG9uZSgpCndoZXJlIGZvcmNlZCBkaXNjYXJkIHN0aWxsIG9jY3VyczogaG93ZXZl ciwgdGhpcyBpcyBsaWtlbHkgdG8gYmUKZW5jb3VudGVyZWQgbGVzcyBmcmVxdWVudGx5IGJ5IGRl YnVnZ2VyIHVzZXJzLgoKVGhpcyBBQkkgY2hhbmdlIHdpbGwgY29tbWl0IHRoZSBrZXJuZWwgdG8g c29tZSBleHRyYSBjb3N0OiBhdCBsZWFzdApzb21lIHRocmVhZCBmbGFnIG1hbmlwdWxhdGlvbiwg Y29uZGl0aW9uYWwgYnJhbmNoaW5nIGFuZAoKCW1vdgl2MC4xNmIsIHYwLjE2YgoJbW92CXYxLjE2 YiwgdjEuMTZiCgkvLyAuLi4KCW1vdiAJdjMxLjE2YiwgdjMxLjE2YgoJcGZhbHNlCXAwLmIKCS8v IC4uLgoJcGZhbHNlCXAxNS5iCgl3cmZmcglwMC5iCgoob3IgZXF1aXZhbGVudCkgb24gdGhlIHN5 c2NhbGwgZmFzdCBwYXRoLCBmb3IgU1ZFIHRhc2tzLgoKRm9yIG5vdyB0aGlzIHBhdGNoIHJlcXVp cmVzIHRyYXAgKyBtZW1zZXQoKSArIHJlbG9hZCwgc28gaXMKc2lnbmlmaWNhbnRseSBtb3JlIGV4 cGVuc2l2ZS4KLS0tCiBhcmNoL2FybTY0L2luY2x1ZGUvYXNtL2Zwc2ltZC5oICAgICAgfCAgMTYg KysKIGFyY2gvYXJtNjQvaW5jbHVkZS9hc20vcHJvY2Vzc29yLmggICB8ICAgMiArCiBhcmNoL2Fy bTY0L2luY2x1ZGUvYXNtL3RocmVhZF9pbmZvLmggfCAgIDQgKwogYXJjaC9hcm02NC9pbmNsdWRl L2FzbS90cmFwcy5oICAgICAgIHwgICAyICsKIGFyY2gvYXJtNjQva2VybmVsL2VudHJ5LlMgICAg ICAgICAgICB8ICAzOSArKysrLQogYXJjaC9hcm02NC9rZXJuZWwvZnBzaW1kLmMgICAgICAgICAg IHwgMzI0ICsrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKystCiBhcmNoL2FybTY0L2tl cm5lbC9wcm9jZXNzLmMgICAgICAgICAgfCAgMjQgKysrCiBhcmNoL2FybTY0L2tlcm5lbC90cmFw cy5jICAgICAgICAgICAgfCAgIDYgKy0KIDggZmlsZXMgY2hhbmdlZCwgNDA2IGluc2VydGlvbnMo KyksIDExIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL2FyY2gvYXJtNjQvaW5jbHVkZS9hc20v ZnBzaW1kLmggYi9hcmNoL2FybTY0L2luY2x1ZGUvYXNtL2Zwc2ltZC5oCmluZGV4IDAyNmE3Yzcu LjU2NTVmZTEgMTAwNjQ0Ci0tLSBhL2FyY2gvYXJtNjQvaW5jbHVkZS9hc20vZnBzaW1kLmgKKysr IGIvYXJjaC9hcm02NC9pbmNsdWRlL2FzbS9mcHNpbWQuaApAQCAtMjAsNiArMjAsOCBAQAogCiAj aWZuZGVmIF9fQVNTRU1CTFlfXwogCisjaW5jbHVkZSA8bGludXgvc3RkZGVmLmg+CisKIC8qCiAg KiBGUC9TSU1EIHN0b3JhZ2UgYXJlYSBoYXM6CiAgKiAgLSBGUFNSIGFuZCBGUENSCkBAIC03Miw2 ICs3NCwyMCBAQCBleHRlcm4gdm9pZCBzdmVfbG9hZF9zdGF0ZSh2b2lkIGNvbnN0ICpzdGF0ZSwg dTMyIGNvbnN0ICpwZnBzciwKIAkJCSAgIHVuc2lnbmVkIGxvbmcgdnFfbWludXNfMSk7CiBleHRl cm4gdW5zaWduZWQgaW50IHN2ZV9nZXRfdmwodm9pZCk7CiAKKyNpZmRlZiBDT05GSUdfQVJNNjRf U1ZFCisKK2V4dGVybiBzaXplX3Qgc3ZlX3N0YXRlX3NpemUoc3RydWN0IHRhc2tfc3RydWN0IGNv bnN0ICp0YXNrKTsKKworZXh0ZXJuIHZvaWQgc3ZlX2FsbG9jKHN0cnVjdCB0YXNrX3N0cnVjdCAq dGFzayk7CitleHRlcm4gdm9pZCBmcHNpbWRfcmVsZWFzZV90YXNrKHN0cnVjdCB0YXNrX3N0cnVj dCAqdGFzayk7CisKKyNlbHNlIC8qICEgQ09ORklHX0FSTTY0X1NWRSAqLworCitzdGF0aWMgaW5s aW5lIHZvaWQgc3ZlX2FsbG9jKHN0cnVjdCB0YXNrX3N0cnVjdCAqdGFzaykgeyB9CitzdGF0aWMg aW5saW5lIHZvaWQgZnBzaW1kX3JlbGVhc2VfdGFzayhzdHJ1Y3QgdGFza19zdHJ1Y3QgKnRhc2sp IHsgfQorCisjZW5kaWYgLyogISBDT05GSUdfQVJNNjRfU1ZFICovCisKIC8qIEZvciB1c2UgYnkg RUZJIHJ1bnRpbWUgc2VydmljZXMgY2FsbHMgb25seSAqLwogZXh0ZXJuIHZvaWQgX19lZmlfZnBz aW1kX2JlZ2luKHZvaWQpOwogZXh0ZXJuIHZvaWQgX19lZmlfZnBzaW1kX2VuZCh2b2lkKTsKZGlm ZiAtLWdpdCBhL2FyY2gvYXJtNjQvaW5jbHVkZS9hc20vcHJvY2Vzc29yLmggYi9hcmNoL2FybTY0 L2luY2x1ZGUvYXNtL3Byb2Nlc3Nvci5oCmluZGV4IDdkZGRjYTIuLmUyZjU3NWQgMTAwNjQ0Ci0t LSBhL2FyY2gvYXJtNjQvaW5jbHVkZS9hc20vcHJvY2Vzc29yLmgKKysrIGIvYXJjaC9hcm02NC9p bmNsdWRlL2FzbS9wcm9jZXNzb3IuaApAQCAtMTA1LDYgKzEwNSw4IEBAIHN0cnVjdCB0aHJlYWRf c3RydWN0IHsKIAl1bnNpZ25lZCBsb25nCQl0cDJfdmFsdWU7CiAjZW5kaWYKIAlzdHJ1Y3QgZnBz aW1kX3N0YXRlCWZwc2ltZF9zdGF0ZTsKKwl2b2lkCQkJKnN2ZV9zdGF0ZTsJLyogU1ZFIHJlZ2lz dGVycywgaWYgYW55ICovCisJdW5zaWduZWQgaW50CQlzdmVfdmw7CQkvKiBTVkUgdmVjdG9yIGxl bmd0aCAqLwogCXVuc2lnbmVkIGxvbmcJCWZhdWx0X2FkZHJlc3M7CS8qIGZhdWx0IGluZm8gKi8K IAl1bnNpZ25lZCBsb25nCQlmYXVsdF9jb2RlOwkvKiBFU1JfRUwxIHZhbHVlICovCiAJc3RydWN0 IGRlYnVnX2luZm8JZGVidWc7CQkvKiBkZWJ1Z2dpbmcgKi8KZGlmZiAtLWdpdCBhL2FyY2gvYXJt NjQvaW5jbHVkZS9hc20vdGhyZWFkX2luZm8uaCBiL2FyY2gvYXJtNjQvaW5jbHVkZS9hc20vdGhy ZWFkX2luZm8uaAppbmRleCBkZGRlZDY0Li45MmI3YjQ4IDEwMDY0NAotLS0gYS9hcmNoL2FybTY0 L2luY2x1ZGUvYXNtL3RocmVhZF9pbmZvLmgKKysrIGIvYXJjaC9hcm02NC9pbmNsdWRlL2FzbS90 aHJlYWRfaW5mby5oCkBAIC02Myw2ICs2Myw4IEBAIHN0cnVjdCB0aHJlYWRfaW5mbyB7CiB2b2lk IGFyY2hfc2V0dXBfbmV3X2V4ZWModm9pZCk7CiAjZGVmaW5lIGFyY2hfc2V0dXBfbmV3X2V4ZWMg ICAgIGFyY2hfc2V0dXBfbmV3X2V4ZWMKIAordm9pZCBhcmNoX3JlbGVhc2VfdGFza19zdHJ1Y3Qo c3RydWN0IHRhc2tfc3RydWN0ICp0c2spOworCiAjZW5kaWYKIAogLyoKQEAgLTkyLDYgKzk0LDcg QEAgdm9pZCBhcmNoX3NldHVwX25ld19leGVjKHZvaWQpOwogI2RlZmluZSBUSUZfUkVTVE9SRV9T SUdNQVNLCTIwCiAjZGVmaW5lIFRJRl9TSU5HTEVTVEVQCQkyMQogI2RlZmluZSBUSUZfMzJCSVQJ CTIyCS8qIDMyYml0IHByb2Nlc3MgKi8KKyNkZWZpbmUgVElGX1NWRQkJCTIzCS8qIFNjYWxhYmxl IFZlY3RvciBFeHRlbnNpb24gaW4gdXNlICovCiAKICNkZWZpbmUgX1RJRl9TSUdQRU5ESU5HCQko MSA8PCBUSUZfU0lHUEVORElORykKICNkZWZpbmUgX1RJRl9ORUVEX1JFU0NIRUQJKDEgPDwgVElG X05FRURfUkVTQ0hFRCkKQEAgLTEwNSw2ICsxMDgsNyBAQCB2b2lkIGFyY2hfc2V0dXBfbmV3X2V4 ZWModm9pZCk7CiAjZGVmaW5lIF9USUZfVVBST0JFCQkoMSA8PCBUSUZfVVBST0JFKQogI2RlZmlu ZSBfVElGX0ZTQ0hFQ0sJCSgxIDw8IFRJRl9GU0NIRUNLKQogI2RlZmluZSBfVElGXzMyQklUCQko MSA8PCBUSUZfMzJCSVQpCisjZGVmaW5lIF9USUZfU1ZFCQkoMSA8PCBUSUZfU1ZFKQogCiAjZGVm aW5lIF9USUZfV09SS19NQVNLCQkoX1RJRl9ORUVEX1JFU0NIRUQgfCBfVElGX1NJR1BFTkRJTkcg fCBcCiAJCQkJIF9USUZfTk9USUZZX1JFU1VNRSB8IF9USUZfRk9SRUlHTl9GUFNUQVRFIHwgXApk aWZmIC0tZ2l0IGEvYXJjaC9hcm02NC9pbmNsdWRlL2FzbS90cmFwcy5oIGIvYXJjaC9hcm02NC9p bmNsdWRlL2FzbS90cmFwcy5oCmluZGV4IDQ1ZTNkYTMuLjE2OTZmOWQgMTAwNjQ0Ci0tLSBhL2Fy Y2gvYXJtNjQvaW5jbHVkZS9hc20vdHJhcHMuaAorKysgYi9hcmNoL2FybTY0L2luY2x1ZGUvYXNt L3RyYXBzLmgKQEAgLTM0LDYgKzM0LDggQEAgc3RydWN0IHVuZGVmX2hvb2sgewogCiB2b2lkIHJl Z2lzdGVyX3VuZGVmX2hvb2soc3RydWN0IHVuZGVmX2hvb2sgKmhvb2spOwogdm9pZCB1bnJlZ2lz dGVyX3VuZGVmX2hvb2soc3RydWN0IHVuZGVmX2hvb2sgKmhvb2spOwordm9pZCBmb3JjZV9zaWdu YWxfaW5qZWN0KGludCBzaWduYWwsIGludCBjb2RlLCBzdHJ1Y3QgcHRfcmVncyAqcmVncywKKwkJ CSB1bnNpZ25lZCBsb25nIGFkZHJlc3MpOwogCiB2b2lkIGFybTY0X25vdGlmeV9zZWdmYXVsdChz dHJ1Y3QgcHRfcmVncyAqcmVncywgdW5zaWduZWQgbG9uZyBhZGRyKTsKIApkaWZmIC0tZ2l0IGEv YXJjaC9hcm02NC9rZXJuZWwvZW50cnkuUyBiL2FyY2gvYXJtNjQva2VybmVsL2VudHJ5LlMKaW5k ZXggZjVlODUxZS4uNTZlODQ4ZiAxMDA2NDQKLS0tIGEvYXJjaC9hcm02NC9rZXJuZWwvZW50cnku UworKysgYi9hcmNoL2FybTY0L2tlcm5lbC9lbnRyeS5TCkBAIC02MDcsNiArNjA3LDggQEAgZWww X3N5bmM6CiAJYi5lcQllbDBfaWEKIAljbXAJeDI0LCAjRVNSX0VMeF9FQ19GUF9BU0lNRAkvLyBG UC9BU0lNRCBhY2Nlc3MKIAliLmVxCWVsMF9mcHNpbWRfYWNjCisJY21wCXgyNCwgI0VTUl9FTHhf RUNfU1ZFCQkvLyBTVkUgYWNjZXNzCisJYi5lcQllbDBfc3ZlX2FjYwogCWNtcAl4MjQsICNFU1Jf RUx4X0VDX0ZQX0VYQzY0CS8vIEZQL0FTSU1EIGV4Y2VwdGlvbgogCWIuZXEJZWwwX2Zwc2ltZF9l eGMKIAljbXAJeDI0LCAjRVNSX0VMeF9FQ19TWVM2NAkJLy8gY29uZmlndXJhYmxlIHRyYXAKQEAg LTY1OCw2ICs2NjAsNyBAQCBlbDBfc3ZjX2NvbXBhdDoKIAkvKgogCSAqIEFBcmNoMzIgc3lzY2Fs bCBoYW5kbGluZwogCSAqLworCWxkcgl4MTYsIFt0c2ssICNUU0tfVElfRkxBR1NdCS8vIGxvYWQg dGhyZWFkIGZsYWdzCiAJYWRycAlzdGJsLCBjb21wYXRfc3lzX2NhbGxfdGFibGUJLy8gbG9hZCBj b21wYXQgc3lzY2FsbCB0YWJsZSBwb2ludGVyCiAJbW92CXdzY25vLCB3NwkJCS8vIHN5c2NhbGwg bnVtYmVyIGluIHc3IChyNykKIAltb3YgICAgIHdzY19uciwgI19fTlJfY29tcGF0X3N5c2NhbGxz CkBAIC03MDUsOSArNzA4LDE5IEBAIGVsMF9mcHNpbWRfYWNjOgogCW1vdgl4MSwgc3AKIAlibAlk b19mcHNpbWRfYWNjCiAJYglyZXRfdG9fdXNlcgorZWwwX3N2ZV9hY2M6CisJLyoKKwkgKiBTY2Fs YWJsZSBWZWN0b3IgRXh0ZW5zaW9uIGFjY2VzcworCSAqLworCWVuYWJsZV9kYmdfYW5kX2lycQor CWN0X3VzZXJfZXhpdAorCW1vdgl4MCwgeDI1CisJbW92CXgxLCBzcAorCWJsCWRvX3N2ZV9hY2MK KwliCXJldF90b191c2VyCiBlbDBfZnBzaW1kX2V4YzoKIAkvKgotCSAqIEZsb2F0aW5nIFBvaW50 IG9yIEFkdmFuY2VkIFNJTUQgZXhjZXB0aW9uCisJICogRmxvYXRpbmcgUG9pbnQsIEFkdmFuY2Vk IFNJTUQgb3IgU1ZFIGV4Y2VwdGlvbgogCSAqLwogCWVuYWJsZV9kYmcKIAljdF91c2VyX2V4aXQK QEAgLTgzNSwxNiArODQ4LDM2IEBAIEVORFBST0MocmV0X3RvX3VzZXIpCiAgKi8KIAkuYWxpZ24J NgogZWwwX3N2YzoKKwlsZHIJeDE2LCBbdHNrLCAjVFNLX1RJX0ZMQUdTXQkvLyBsb2FkIHRocmVh ZCBmbGFncwogCWFkcnAJc3RibCwgc3lzX2NhbGxfdGFibGUJCS8vIGxvYWQgc3lzY2FsbCB0YWJs ZSBwb2ludGVyCiAJbW92CXdzY25vLCB3OAkJCS8vIHN5c2NhbGwgbnVtYmVyIGluIHc4CiAJbW92 CXdzY19uciwgI19fTlJfc3lzY2FsbHMKKworI2lmbmRlZiBDT05GSUdfQVJNNjRfU1ZFCisJYgll bDBfc3ZjX25ha2VkCisjZWxzZQorCXRiegl4MTYsICNUSUZfU1ZFLCBlbDBfc3ZjX25ha2VkCS8v IFNraXAgdW5sZXNzIFRJRl9TVkUgc2V0OgorCWJpYwl4MTYsIHgxNiwgI19USUZfU1ZFCQkvLyBk aXNjYXJkIFNWRSBzdGF0ZQorCXN0cgl4MTYsIFt0c2ssICNUU0tfVElfRkxBR1NdCisKKwkvKgor CSAqIHRhc2tfZnBzaW1kX2xvYWQoKSB3b24ndCBiZSBjYWxsZWQgdG8gdXBkYXRlIENQQUNSX0VM MSBpbgorCSAqIHJldF90b191c2VyIHVubGVzcyBUSUZfRk9SRUlHTl9GUFNUQVRFIGlzIHN0aWxs IHNldCwgd2hpY2ggb25seQorCSAqIGhhcHBlbnMgaWYgYSBjb250ZXh0IHN3aXRjaCBvciBrZXJu ZWxfbmVvbl9iZWdpbigpIG9yIGNvbnRleHQKKwkgKiBtb2RpZmljYXRpb24gKHNpZ3JldHVybiwg cHRyYWNlKSBpbnRlcnZlbmVzLgorCSAqIFNvLCBlbnN1cmUgdGhhdCBDUEFDUl9FTDEgaXMgYWxy ZWFkeSBjb3JyZWN0IGZvciB0aGUgZmFzdC1wYXRoIGNhc2U6CisJICovCisJbXJzCXg5LCBjcGFj cl9lbDEKKwliaWMJeDksIHg5LCAjQ1BBQ1JfRUwxX1pFTl9FTDBFTgkvLyBkaXNhYmxlIFNWRSBm b3IgZWwwCisJbXNyCWNwYWNyX2VsMSwgeDkJCQkvLyBzeW5jaHJvbmlzZWQgYnkgZXJldCB0byBl bDAKKyNlbmRpZiAvKiBDT05GSUdfQVJNNjRfU1ZFICovCisKIGVsMF9zdmNfbmFrZWQ6CQkJCQkv LyBjb21wYXQgZW50cnkgcG9pbnQKIAlzdHAJeDAsIHhzY25vLCBbc3AsICNTX09SSUdfWDBdCS8v IHNhdmUgdGhlIG9yaWdpbmFsIHgwIGFuZCBzeXNjYWxsIG51bWJlcgogCWVuYWJsZV9kYmdfYW5k X2lycQogCWN0X3VzZXJfZXhpdCAxCiAKLQlsZHIJeDE2LCBbdHNrLCAjVFNLX1RJX0ZMQUdTXQkv LyBjaGVjayBmb3Igc3lzY2FsbCBob29rcwotCXRzdAl4MTYsICNfVElGX1NZU0NBTExfV09SSwor CXRzdAl4MTYsICNfVElGX1NZU0NBTExfV09SSwkJLy8gY2hlY2sgZm9yIHN5c2NhbGwgaG9va3MK IAliLm5lCV9fc3lzX3RyYWNlCiAJY21wICAgICB3c2Nubywgd3NjX25yCQkJLy8gY2hlY2sgdXBw ZXIgc3lzY2FsbCBsaW1pdAogCWIuaHMJbmlfc3lzCmRpZmYgLS1naXQgYS9hcmNoL2FybTY0L2tl cm5lbC9mcHNpbWQuYyBiL2FyY2gvYXJtNjQva2VybmVsL2Zwc2ltZC5jCmluZGV4IDdhODY1Yzgu LjBiMTVhNTk0IDEwMDY0NAotLS0gYS9hcmNoL2FybTY0L2tlcm5lbC9mcHNpbWQuYworKysgYi9h cmNoL2FybTY0L2tlcm5lbC9mcHNpbWQuYwpAQCAtMTgsMTggKzE4LDI2IEBACiAgKi8KIAogI2lu Y2x1ZGUgPGxpbnV4L2JvdHRvbV9oYWxmLmg+CisjaW5jbHVkZSA8bGludXgvYnVnLmg+CisjaW5j bHVkZSA8bGludXgvY29tcGF0Lmg+CiAjaW5jbHVkZSA8bGludXgvY3B1Lmg+CiAjaW5jbHVkZSA8 bGludXgvY3B1X3BtLmg+CiAjaW5jbHVkZSA8bGludXgva2VybmVsLmg+CisjaW5jbHVkZSA8bGlu dXgvaXJxZmxhZ3MuaD4KICNpbmNsdWRlIDxsaW51eC9pbml0Lmg+CiAjaW5jbHVkZSA8bGludXgv cGVyY3B1Lmg+CiAjaW5jbHVkZSA8bGludXgvcHJlZW1wdC5oPgorI2luY2x1ZGUgPGxpbnV4L3B0 cmFjZS5oPgogI2luY2x1ZGUgPGxpbnV4L3NjaGVkL3NpZ25hbC5oPgogI2luY2x1ZGUgPGxpbnV4 L3NpZ25hbC5oPgorI2luY2x1ZGUgPGxpbnV4L3NsYWIuaD4KIAogI2luY2x1ZGUgPGFzbS9mcHNp bWQuaD4KICNpbmNsdWRlIDxhc20vY3B1dHlwZS5oPgogI2luY2x1ZGUgPGFzbS9zaW1kLmg+Cisj aW5jbHVkZSA8YXNtL3NpZ2NvbnRleHQuaD4KKyNpbmNsdWRlIDxhc20vc3lzcmVnLmg+CisjaW5j bHVkZSA8YXNtL3RyYXBzLmg+CiAKICNkZWZpbmUgRlBFWENfSU9GCSgxIDw8IDApCiAjZGVmaW5l IEZQRVhDX0RaRgkoMSA8PCAxKQpAQCAtMzksNiArNDcsOCBAQAogI2RlZmluZSBGUEVYQ19JREYJ KDEgPDwgNykKIAogLyoKKyAqIChOb3RlOiBpbiB0aGlzIGRpc2N1c3Npb24sIHN0YXRlbWVudHMg YWJvdXQgRlBTSU1EIGFwcGx5IGVxdWFsbHkgdG8gU1ZFLikKKyAqCiAgKiBJbiBvcmRlciB0byBy ZWR1Y2UgdGhlIG51bWJlciBvZiB0aW1lcyB0aGUgRlBTSU1EIHN0YXRlIGlzIG5lZWRsZXNzbHkg c2F2ZWQKICAqIGFuZCByZXN0b3JlZCwgd2UgbmVlZCB0byBrZWVwIHRyYWNrIG9mIHR3byB0aGlu Z3M6CiAgKiAoYSkgZm9yIGVhY2ggdGFzaywgd2UgbmVlZCB0byByZW1lbWJlciB3aGljaCBDUFUg d2FzIHRoZSBsYXN0IG9uZSB0byBoYXZlCkBAIC0xMDAsNiArMTEwLDI3OSBAQAogc3RhdGljIERF RklORV9QRVJfQ1BVKHN0cnVjdCBmcHNpbWRfc3RhdGUgKiwgZnBzaW1kX2xhc3Rfc3RhdGUpOwog CiAvKgorICogQ2FsbCBfX3N2ZV9mcmVlKCkgZGlyZWN0bHkgb25seSBpZiB5b3Uga25vdyB0YXNr IGNhbid0IGJlIHNjaGVkdWxlZAorICogb3IgcHJlZW1wdGVkLgorICovCitzdGF0aWMgdm9pZCBf X3N2ZV9mcmVlKHN0cnVjdCB0YXNrX3N0cnVjdCAqdGFzaykKK3sKKwlrZnJlZSh0YXNrLT50aHJl YWQuc3ZlX3N0YXRlKTsKKwl0YXNrLT50aHJlYWQuc3ZlX3N0YXRlID0gTlVMTDsKK30KKworc3Rh dGljIHZvaWQgc3ZlX2ZyZWUoc3RydWN0IHRhc2tfc3RydWN0ICp0YXNrKQoreworCVdBUk5fT04o dGVzdF90c2tfdGhyZWFkX2ZsYWcodGFzaywgVElGX1NWRSkpOworCisJX19zdmVfZnJlZSh0YXNr KTsKK30KKworCisvKiBPZmZzZXQgb2YgRkZSIGluIHRoZSBTVkUgcmVnaXN0ZXIgZHVtcCAqLwor c3RhdGljIHNpemVfdCBzdmVfZmZyX29mZnNldChpbnQgdmwpCit7CisJcmV0dXJuIFNWRV9TSUdf RkZSX09GRlNFVChzdmVfdnFfZnJvbV92bCh2bCkpIC0gU1ZFX1NJR19SRUdTX09GRlNFVDsKK30K Kworc3RhdGljIHZvaWQgKnN2ZV9wZmZyKHN0cnVjdCB0YXNrX3N0cnVjdCAqdGFzaykKK3sKKwly ZXR1cm4gKGNoYXIgKil0YXNrLT50aHJlYWQuc3ZlX3N0YXRlICsKKwkJc3ZlX2Zmcl9vZmZzZXQo dGFzay0+dGhyZWFkLnN2ZV92bCk7Cit9CisKK3N0YXRpYyB2b2lkIGNoYW5nZV9jcGFjcih1NjQg dmFsLCB1NjQgbWFzaykKK3sKKwl1NjQgY3BhY3IgPSByZWFkX3N5c3JlZyhDUEFDUl9FTDEpOwor CXU2NCBuZXcgPSAoY3BhY3IgJiB+bWFzaykgfCB2YWw7CisKKwlpZiAobmV3ICE9IGNwYWNyKQor CQl3cml0ZV9zeXNyZWcobmV3LCBDUEFDUl9FTDEpOworfQorCitzdGF0aWMgdm9pZCBzdmVfdXNl cl9kaXNhYmxlKHZvaWQpCit7CisJY2hhbmdlX2NwYWNyKDAsIENQQUNSX0VMMV9aRU5fRUwwRU4p OworfQorCitzdGF0aWMgdm9pZCBzdmVfdXNlcl9lbmFibGUodm9pZCkKK3sKKwljaGFuZ2VfY3Bh Y3IoQ1BBQ1JfRUwxX1pFTl9FTDBFTiwgQ1BBQ1JfRUwxX1pFTl9FTDBFTik7Cit9CisKKy8qCisg KiBUSUZfU1ZFIGNvbnRyb2xzIHdoZXRoZXIgYSB0YXNrIGNhbiB1c2UgU1ZFIHdpdGhvdXQgdHJh cHBpbmcgd2hpbGUKKyAqIGluIHVzZXJzcGFjZSwgYW5kIGFsc28gdGhlIHdheSBhIHRhc2sncyBG UFNJTUQvU1ZFIHN0YXRlIGlzIHN0b3JlZAorICogaW4gdGhyZWFkX3N0cnVjdC4KKyAqCisgKiBU aGUga2VybmVsIHVzZXMgdGhpcyBmbGFnIHRvIHRyYWNrIHdoZXRoZXIgYSB1c2VyIHRhc2sgaXMg YWN0aXZlbHkKKyAqIHVzaW5nIFNWRSwgYW5kIHRoZXJlZm9yZSB3aGV0aGVyIGZ1bGwgU1ZFIHJl Z2lzdGVyIHN0YXRlIG5lZWRzIHRvCisgKiBiZSB0cmFja2VkLiAgSWYgbm90LCB0aGUgY2hlYXBl ciBGUFNJTUQgY29udGV4dCBoYW5kbGluZyBjb2RlIGNhbgorICogYmUgdXNlZCBpbnN0ZWFkIG9m IHRoZSBtb3JlIGNvc3RseSBTVkUgZXF1aXZhbGVudHMuCisgKgorICogICogVElGX1NWRSBzZXQ6 CisgKgorICogICAgVGhlIHRhc2sgY2FuIGV4ZWN1dGUgU1ZFIGluc3RydWN0aW9ucyB3aGlsZSBp biB1c2Vyc3BhY2Ugd2l0aG91dAorICogICAgdHJhcHBpbmcgdG8gdGhlIGtlcm5lbC4KKyAqCisg KiAgICBXaGVuIHN0b3JlZCwgWjAtWjMxIChpbmNvcnBvcmF0aW5nIFZuIGluIGJpdHNbMTI3OjBd IG9yIHRoZQorICogICAgY29ycmVzcG9uZGluZyBabiksIFAwLVAxNSBhbmQgRkZSIGFyZSBlbmNv ZGVkIGluIGluCisgKiAgICB0YXNrLT50aHJlYWQuc3ZlX3N0YXRlLCBmb3JtYXR0ZWQgYXBwcm9w cmlhdGVseSBmb3IgdmVjdG9yCisgKiAgICBsZW5ndGggdGFzay0+dGhyZWFkLnN2ZV92bC4KKyAq CisgKiAgICB0YXNrLT50aHJlYWQuc3ZlX3N0YXRlIG11c3QgcG9pbnQgdG8gYSB2YWxpZCBidWZm ZXIgYXQgbGVhc3QKKyAqICAgIHN2ZV9zdGF0ZV9zaXplKHRhc2spIGJ5dGVzIGluIHNpemUuCisg KgorICogICAgRHVyaW5nIGFueSBzeXNjYWxsLCB0aGUga2VybmVsIG1heSBvcHRpb25hbGx5IGNs ZWFyIFRJRl9TVkUgYW5kCisgKiAgICBkaXNjYXJkIHRoZSB2ZWN0b3Igc3RhdGUgZXhjZXB0IGZv ciB0aGUgRlBTSU1EIHN1YnNldC4KKyAqCisgKiAgKiBUSUZfU1ZFIGNsZWFyOgorICoKKyAqICAg IEFuIGF0dGVtcHQgYnkgdGhlIHVzZXIgdGFzayB0byBleGVjdXRlIGFuIFNWRSBpbnN0cnVjdGlv biBjYXVzZXMKKyAqICAgIGRvX3N2ZV9hY2MoKSB0byBiZSBjYWxsZWQsIHdoaWNoIGRvZXMgc29t ZSBwcmVwYXJhdGlvbiBhbmQgdGhlbgorICogICAgc2V0cyBUSUZfU1ZFLgorICoKKyAqICAgIFdo ZW4gc3RvcmVkLCBGUFNJTUQgcmVnaXN0ZXJzIFYwLVYzMSBhcmUgZW5jb2RlZCBpbgorICogICAg dGFzay0+ZnBzaW1kX3N0YXRlOyBiaXRzIFttYXggOiAxMjhdIGZvciBlYWNoIG9mIFowLVozMSBh cmUKKyAqICAgIGxvZ2ljYWxseSB6ZXJvIGJ1dCBub3Qgc3RvcmVkIGFueXdoZXJlOyBQMC1QMTUg YW5kIEZGUiBhcmUgbm90CisgKiAgICBzdG9yZWQgYW5kIGhhdmUgdW5zcGVjaWZpZWQgdmFsdWVz IGZyb20gdXNlcnNwYWNlJ3MgcG9pbnQgb2YKKyAqICAgIHZpZXcuICBGb3IgaHlnaWVuZSBwdXJw b3NlcywgdGhlIGtlcm5lbCB6ZXJvZXMgdGhlbSBvbiBuZXh0IHVzZSwKKyAqICAgIGJ1dCB1c2Vy c3BhY2UgaXMgZGlzY291cmFnZWQgZnJvbSByZWx5aW5nIG9uIHRoaXMuCisgKgorICogICAgdGFz ay0+dGhyZWFkLnN2ZV9zdGF0ZSBkb2VzIG5vdCBuZWVkIHRvIGJlIG5vbi1OVUxMLCB2YWxpZCBv ciBhbnkKKyAqICAgIHBhcnRpY3VsYXIgc2l6ZTogaXQgbXVzdCBub3QgYmUgZGVyZWZlcmVuY2Vk LgorICoKKyAqICAqIEZQU1IgYW5kIEZQQ1IgYXJlIGFsd2F5cyBzdG9yZWQgaW4gdGFzay0+ZnBz aW1kX3N0YXRlIGlycmVzcGN0aXZlIG9mCisgKiAgICB3aGV0aGVyIFRJRl9TVkUgaXMgY2xlYXIg b3Igc2V0LCBzaW5jZSB0aGVzZSBhcmUgbm90IHZlY3RvciBsZW5ndGgKKyAqICAgIGRlcGVuZGVu dC4KKyAqLworCisvKgorICogVXBkYXRlIGN1cnJlbnQncyBGUFNJTUQvU1ZFIHJlZ2lzdGVycyBm cm9tIHRocmVhZF9zdHJ1Y3QuCisgKgorICogVGhpcyBmdW5jdGlvbiBzaG91bGQgYmUgY2FsbGVk IG9ubHkgd2hlbiB0aGUgRlBTSU1EL1NWRSBzdGF0ZSBpbgorICogdGhyZWFkX3N0cnVjdCBpcyBr bm93biB0byBiZSB1cCB0byBkYXRlLCB3aGVuIHByZXBhcmluZyB0byBlbnRlcgorICogdXNlcnNw YWNlLgorICoKKyAqIFNvZnRpcnFzIChhbmQgcHJlZW1wdGlvbikgbXVzdCBiZSBkaXNhYmxlZC4K KyAqLworc3RhdGljIHZvaWQgdGFza19mcHNpbWRfbG9hZCh2b2lkKQoreworCVdBUk5fT04oIWlu X3NvZnRpcnEoKSAmJiAhaXJxc19kaXNhYmxlZCgpKTsKKworCWlmIChzeXN0ZW1fc3VwcG9ydHNf c3ZlKCkgJiYgdGVzdF90aHJlYWRfZmxhZyhUSUZfU1ZFKSkKKwkJc3ZlX2xvYWRfc3RhdGUoc3Zl X3BmZnIoY3VycmVudCksCisJCQkgICAgICAgJmN1cnJlbnQtPnRocmVhZC5mcHNpbWRfc3RhdGUu ZnBzciwKKwkJCSAgICAgICBzdmVfdnFfZnJvbV92bChjdXJyZW50LT50aHJlYWQuc3ZlX3ZsKSAt IDEpOworCWVsc2UKKwkJZnBzaW1kX2xvYWRfc3RhdGUoJmN1cnJlbnQtPnRocmVhZC5mcHNpbWRf c3RhdGUpOworCisJaWYgKHN5c3RlbV9zdXBwb3J0c19zdmUoKSkgeworCQkvKiBUb2dnbGUgU1ZF IHRyYXBwaW5nIGZvciB1c2Vyc3BhY2UgaWYgbmVlZGVkICovCisJCWlmICh0ZXN0X3RocmVhZF9m bGFnKFRJRl9TVkUpKQorCQkJc3ZlX3VzZXJfZW5hYmxlKCk7CisJCWVsc2UKKwkJCXN2ZV91c2Vy X2Rpc2FibGUoKTsKKworCQkvKiBTZXJpYWxpc2VkIGJ5IGV4Y2VwdGlvbiByZXR1cm4gdG8gdXNl ciAqLworCX0KK30KKworLyoKKyAqIEVuc3VyZSBjdXJyZW50J3MgRlBTSU1EL1NWRSBzdG9yYWdl IGluIHRocmVhZF9zdHJ1Y3QgaXMgdXAgdG8gZGF0ZQorICogd2l0aCByZXNwZWN0IHRvIHRoZSBD UFUgcmVnaXN0ZXJzLgorICoKKyAqIFNvZnRpcnFzIChhbmQgcHJlZW1wdGlvbikgbXVzdCBiZSBk aXNhYmxlZC4KKyAqLworc3RhdGljIHZvaWQgdGFza19mcHNpbWRfc2F2ZSh2b2lkKQoreworCVdB Uk5fT04oIWluX3NvZnRpcnEoKSAmJiAhaXJxc19kaXNhYmxlZCgpKTsKKworCWlmICghdGVzdF90 aHJlYWRfZmxhZyhUSUZfRk9SRUlHTl9GUFNUQVRFKSkgeworCQlpZiAoc3lzdGVtX3N1cHBvcnRz X3N2ZSgpICYmIHRlc3RfdGhyZWFkX2ZsYWcoVElGX1NWRSkpIHsKKwkJCWlmIChXQVJOX09OKHN2 ZV9nZXRfdmwoKSAhPSBjdXJyZW50LT50aHJlYWQuc3ZlX3ZsKSkgeworCQkJCS8qCisJCQkJICog Q2FuJ3Qgc2F2ZSB0aGUgdXNlciByZWdzLCBzbyBjdXJyZW50IHdvdWxkCisJCQkJICogcmUtZW50 ZXIgdXNlciB3aXRoIGNvcnJ1cHQgc3RhdGUuCisJCQkJICogVGhlcmUncyBubyB3YXkgdG8gcmVj b3Zlciwgc28ga2lsbCBpdDoKKwkJCQkgKi8KKwkJCQlmb3JjZV9zaWduYWxfaW5qZWN0KAorCQkJ CQlTSUdLSUxMLCAwLCBjdXJyZW50X3B0X3JlZ3MoKSwgMCk7CisJCQkJcmV0dXJuOworCQkJfQor CisJCQlzdmVfc2F2ZV9zdGF0ZShzdmVfcGZmcihjdXJyZW50KSwKKwkJCQkgICAgICAgJmN1cnJl bnQtPnRocmVhZC5mcHNpbWRfc3RhdGUuZnBzcik7CisJCX0gZWxzZQorCQkJZnBzaW1kX3NhdmVf c3RhdGUoJmN1cnJlbnQtPnRocmVhZC5mcHNpbWRfc3RhdGUpOworCX0KK30KKworI2RlZmluZSBa UkVHKHN2ZV9zdGF0ZSwgdnEsIG4pICgoY2hhciAqKShzdmVfc3RhdGUpICsJCVwKKwkoU1ZFX1NJ R19aUkVHX09GRlNFVCh2cSwgbikgLSBTVkVfU0lHX1JFR1NfT0ZGU0VUKSkKKworLyoKKyAqIFRy YW5zZmVyIHRoZSBGUFNJTUQgc3RhdGUgaW4gdGFzay0+dGhyZWFkLmZwc2ltZF9zdGF0ZSB0bwor ICogdGFzay0+dGhyZWFkLnN2ZV9zdGF0ZS4KKyAqCisgKiBUYXNrIGNhbiBiZSBhIG5vbi1ydW5u YWJsZSB0YXNrLCBvciBjdXJyZW50LiAgSW4gdGhlIGxhdHRlciBjYXNlLAorICogc29mdGlycXMg KGFuZCBwcmVlbXB0aW9uKSBtdXN0IGJlIGRpc2FibGVkLgorICogdGFzay0+dGhyZWFkLnN2ZV9z dGF0ZSBtdXN0IHBvaW50IHRvIGF0IGxlYXN0IHN2ZV9zdGF0ZV9zaXplKHRhc2spCisgKiBieXRl cyBvZiBhbGxvY2F0ZWQga2VybmVsIG1lbW9yeS4KKyAqIHRhc2stPnRocmVhZC5mcHNpbWRfc3Rh dGUgbXVzdCBiZSB1cCB0byBkYXRlIGJlZm9yZSBjYWxsaW5nIHRoaXMgZnVuY3Rpb24uCisgKi8K K3N0YXRpYyB2b2lkIGZwc2ltZF90b19zdmUoc3RydWN0IHRhc2tfc3RydWN0ICp0YXNrKQorewor CXVuc2lnbmVkIGludCB2cTsKKwl2b2lkICpzc3QgPSB0YXNrLT50aHJlYWQuc3ZlX3N0YXRlOwor CXN0cnVjdCBmcHNpbWRfc3RhdGUgY29uc3QgKmZzdCA9ICZ0YXNrLT50aHJlYWQuZnBzaW1kX3N0 YXRlOworCXVuc2lnbmVkIGludCBpOworCisJaWYgKCFzeXN0ZW1fc3VwcG9ydHNfc3ZlKCkpCisJ CXJldHVybjsKKworCXZxID0gc3ZlX3ZxX2Zyb21fdmwodGFzay0+dGhyZWFkLnN2ZV92bCk7CisJ Zm9yIChpID0gMDsgaSA8IDMyOyArK2kpCisJCW1lbWNweShaUkVHKHNzdCwgdnEsIGkpLCAmZnN0 LT52cmVnc1tpXSwKKwkJICAgICAgIHNpemVvZihmc3QtPnZyZWdzW2ldKSk7Cit9CisKKyNpZmRl ZiBDT05GSUdfQVJNNjRfU1ZFCisKKy8qCisgKiBSZXR1cm4gaG93IG1hbnkgYnl0ZXMgb2YgbWVt b3J5IGFyZSByZXF1aXJlZCB0byBzdG9yZSB0aGUgZnVsbCBTVkUKKyAqIHN0YXRlIGZvciB0YXNr LCBnaXZlbiB0YXNrJ3MgY3VycmVudGx5IGNvbmZpZ3VyZWQgdmVjdG9yIGxlbmd0aC4KKyAqLwor c2l6ZV90IHN2ZV9zdGF0ZV9zaXplKHN0cnVjdCB0YXNrX3N0cnVjdCBjb25zdCAqdGFzaykKK3sK KwlyZXR1cm4gU1ZFX1NJR19SRUdTX1NJWkUoc3ZlX3ZxX2Zyb21fdmwodGFzay0+dGhyZWFkLnN2 ZV92bCkpOworfQorCisvKgorICogRW5zdXJlIHRoYXQgdGFzay0+dGhyZWFkLnN2ZV9zdGF0ZSBp cyBhbGxvY2F0ZWQgYW5kIHN1ZmZpY2llbnRseSBsYXJnZS4KKyAqCisgKiBUaGlzIGZ1bmN0aW9u IHNob3VsZCBiZSB1c2VkIG9ubHkgaW4gcHJlcGFyYXRpb24gZm9yIHJlcGxhY2luZworICogdGFz ay0+dGhyZWFkLnN2ZV9zdGF0ZSB3aXRoIG5ldyBkYXRhLiAgVGhlIG1lbW9yeSBpcyBhbHdheXMg emVyb2VkCisgKiBoZXJlIHRvIHByZXZlbnQgc3RhbGUgZGF0YSBmcm9tIHNob3dpbmcgdGhyb3Vn aDogdGhpcyBpcyBkb25lIGluCisgKiB0aGUgaW50ZXJlc3Qgb2YgdGVzdGFiaWxpdHkgYW5kIHBy ZWRpY3RhYmlsaXR5OiBleGNlcHQgaW4gdGhlCisgKiBkb19zdmVfYWNjKCkgY2FzZSwgdGhlcmUg aXMgbm8gQUJJIHJlcXVpcmVtZW50IHRvIGhpZGUgc3RhbGUgZGF0YQorICogd3JpdHRlbiBwcmV2 aW91c2x5IGJlIHRhc2suCisgKi8KK3ZvaWQgc3ZlX2FsbG9jKHN0cnVjdCB0YXNrX3N0cnVjdCAq dGFzaykKK3sKKwlpZiAodGFzay0+dGhyZWFkLnN2ZV9zdGF0ZSkgeworCQltZW1zZXQodGFzay0+ dGhyZWFkLnN2ZV9zdGF0ZSwgMCwgc3ZlX3N0YXRlX3NpemUoY3VycmVudCkpOworCQlyZXR1cm47 CisJfQorCisJLyogVGhpcyBpcyBhIHNtYWxsIGFsbG9jYXRpb24gKG1heGltdW0gfjhLQikgYW5k IFNob3VsZCBOb3QgRmFpbC4gKi8KKwl0YXNrLT50aHJlYWQuc3ZlX3N0YXRlID0KKwkJa3phbGxv YyhzdmVfc3RhdGVfc2l6ZSh0YXNrKSwgR0ZQX0tFUk5FTCk7CisKKwkvKgorCSAqIElmIGZ1dHVy ZSBTVkUgcmV2aXNpb25zIGNhbiBoYXZlIGxhcmdlciB2ZWN0b3JzIHRob3VnaCwKKwkgKiB0aGlz IG1heSBjZWFzZSB0byBiZSB0cnVlOgorCSAqLworCUJVR19PTighdGFzay0+dGhyZWFkLnN2ZV9z dGF0ZSk7Cit9CisKKy8qCisgKiBDYWxsZWQgZnJvbSB0aGUgcHV0X3Rhc2tfc3RydWN0KCkgcGF0 aCwgd2hpY2ggY2Fubm90IGdldCBoZXJlCisgKiB1bmxlc3MgZGVhZF90YXNrIGlzIHJlYWxseSBk ZWFkIGFuZCBub3Qgc2NoZWR1bGFibGUuCisgKi8KK3ZvaWQgZnBzaW1kX3JlbGVhc2VfdGFzayhz dHJ1Y3QgdGFza19zdHJ1Y3QgKmRlYWRfdGFzaykKK3sKKwlfX3N2ZV9mcmVlKGRlYWRfdGFzayk7 Cit9CisKKyNlbmRpZiAvKiBDT05GSUdfQVJNNjRfU1ZFICovCisKKy8qCisgKiBUcmFwcGVkIFNW RSBhY2Nlc3MKKyAqCisgKiBTdG9yYWdlIGlzIGFsbG9jYXRlZCBmb3IgdGhlIGZ1bGwgU1ZFIHN0 YXRlLCB0aGUgY3VycmVudCBGUFNJTUQKKyAqIHJlZ2lzdGVyIGNvbnRlbnRzIGFyZSBtaWdyYXRl ZCBhY3Jvc3MsIGFuZCBUSUZfU1ZFIGlzIHNldCBzbyB0aGF0CisgKiB0aGUgU1ZFIGFjY2VzcyB0 cmFwIHdpbGwgYmUgZGlzYWJsZWQgdGhlIG5leHQgdGltZSB0aGlzIHRhc2sKKyAqIHJlYWNoZXMg cmV0X3RvX3VzZXIuCisgKgorICogVElGX1NWRSBzaG91bGQgYmUgY2xlYXIgb24gZW50cnk6IG90 aGVyd2lzZSwgdGFza19mcHNpbWRfbG9hZCgpCisgKiB3b3VsZCBoYXZlIGRpc2FibGVkIHRoZSBT VkUgYWNjZXNzIHRyYXAgZm9yIHVzZXJzcGFjZSBkdXJpbmcKKyAqIHJldF90b191c2VyLCBtYWtp bmcgYW4gU1ZFIGFjY2VzcyB0cmFwIGltcG9zc2libGUgaW4gdGhhdCBjYXNlLgorICovCit2b2lk IGRvX3N2ZV9hY2ModW5zaWduZWQgaW50IGVzciwgc3RydWN0IHB0X3JlZ3MgKnJlZ3MpCit7CisJ LyogRXZlbiBpZiB3ZSBjaG9zZSBub3QgdG8gdXNlIFNWRSwgdGhlIGhhcmR3YXJlIGNvdWxkIHN0 aWxsIHRyYXA6ICovCisJaWYgKHVubGlrZWx5KCFzeXN0ZW1fc3VwcG9ydHNfc3ZlKCkpIHx8IFdB Uk5fT04oaXNfY29tcGF0X3Rhc2soKSkpIHsKKwkJZm9yY2Vfc2lnbmFsX2luamVjdChTSUdJTEws IElMTF9JTExPUEMsIHJlZ3MsIDApOworCQlyZXR1cm47CisJfQorCisJc3ZlX2FsbG9jKGN1cnJl bnQpOworCisJbG9jYWxfYmhfZGlzYWJsZSgpOworCisJdGFza19mcHNpbWRfc2F2ZSgpOworCWZw c2ltZF90b19zdmUoY3VycmVudCk7CisKKwkvKiBGb3JjZSByZXRfdG9fdXNlciB0byByZWxvYWQg dGhlIHJlZ2lzdGVyczogKi8KKwlmcHNpbWRfZmx1c2hfdGFza19zdGF0ZShjdXJyZW50KTsKKwlz ZXRfdGhyZWFkX2ZsYWcoVElGX0ZPUkVJR05fRlBTVEFURSk7CisKKwlpZiAodGVzdF9hbmRfc2V0 X3RocmVhZF9mbGFnKFRJRl9TVkUpKQorCQlXQVJOX09OKDEpOyAvKiBTVkUgYWNjZXNzIHNob3Vs ZG4ndCBoYXZlIHRyYXBwZWQgKi8KKworCWxvY2FsX2JoX2VuYWJsZSgpOworfQorCisvKgogICog VHJhcHBlZCBGUC9BU0lNRCBhY2Nlc3MuCiAgKi8KIHZvaWQgZG9fZnBzaW1kX2FjYyh1bnNpZ25l ZCBpbnQgZXNyLCBzdHJ1Y3QgcHRfcmVncyAqcmVncykKQEAgLTE0NCw4ICs0MjcsOCBAQCB2b2lk IGZwc2ltZF90aHJlYWRfc3dpdGNoKHN0cnVjdCB0YXNrX3N0cnVjdCAqbmV4dCkKIAkgKiB0aGUg cmVnaXN0ZXJzIGlzIGluIGZhY3QgdGhlIG1vc3QgcmVjZW50IHVzZXJsYW5kIEZQU0lNRCBzdGF0 ZSBvZgogCSAqICdjdXJyZW50Jy4KIAkgKi8KLQlpZiAoY3VycmVudC0+bW0gJiYgIXRlc3RfdGhy ZWFkX2ZsYWcoVElGX0ZPUkVJR05fRlBTVEFURSkpCi0JCWZwc2ltZF9zYXZlX3N0YXRlKCZjdXJy ZW50LT50aHJlYWQuZnBzaW1kX3N0YXRlKTsKKwlpZiAoY3VycmVudC0+bW0pCisJCXRhc2tfZnBz aW1kX3NhdmUoKTsKIAogCWlmIChuZXh0LT5tbSkgewogCQkvKgpAQCAtMTY3LDYgKzQ1MCw4IEBA IHZvaWQgZnBzaW1kX3RocmVhZF9zd2l0Y2goc3RydWN0IHRhc2tfc3RydWN0ICpuZXh0KQogCiB2 b2lkIGZwc2ltZF9mbHVzaF90aHJlYWQodm9pZCkKIHsKKwlpbnQgdmw7CisKIAlpZiAoIXN5c3Rl bV9zdXBwb3J0c19mcHNpbWQoKSkKIAkJcmV0dXJuOwogCkBAIC0xNzQsNiArNDU5LDMwIEBAIHZv aWQgZnBzaW1kX2ZsdXNoX3RocmVhZCh2b2lkKQogCiAJbWVtc2V0KCZjdXJyZW50LT50aHJlYWQu ZnBzaW1kX3N0YXRlLCAwLCBzaXplb2Yoc3RydWN0IGZwc2ltZF9zdGF0ZSkpOwogCWZwc2ltZF9m bHVzaF90YXNrX3N0YXRlKGN1cnJlbnQpOworCisJaWYgKHN5c3RlbV9zdXBwb3J0c19zdmUoKSkg eworCQljbGVhcl90aHJlYWRfZmxhZyhUSUZfU1ZFKTsKKwkJc3ZlX2ZyZWUoY3VycmVudCk7CisK KwkJLyoKKwkJICogUmVzZXQgdGhlIHRhc2sgdmVjdG9yIGxlbmd0aCBhcyByZXF1aXJlZC4KKwkJ ICogVGhpcyBpcyB3aGVyZSB3ZSBlbnN1cmUgdGhhdCBhbGwgdXNlciB0YXNrcyBoYXZlIGEgdmFs aWQKKwkJICogdmVjdG9yIGxlbmd0aCBjb25maWd1cmVkOiBubyBrZXJuZWwgdGFzayBjYW4gYmVj b21lIGEgdXNlcgorCQkgKiB0YXNrIHdpdGhvdXQgYW4gZXhlYyBhbmQgaGVuY2UgYSBjYWxsIHRv IHRoaXMgZnVuY3Rpb24uCisJCSAqIElmIGEgYnVnIGNhdXNlcyB0aGlzIHRvIGdvIHdyb25nLCB3 ZSBtYWtlIHNvbWUgbm9pc2UgYW5kCisJCSAqIHRyeSB0byBmdWRnZSB0aHJlYWQuc3ZlX3ZsIHRv IGEgc2FmZSB2YWx1ZSBoZXJlLgorCQkgKi8KKwkJdmwgPSBjdXJyZW50LT50aHJlYWQuc3ZlX3Zs OworCisJCWlmICh2bCA9PSAwKQorCQkJdmwgPSBTVkVfVkxfTUlOOworCisJCWlmIChXQVJOX09O KCFzdmVfdmxfdmFsaWQodmwpKSkKKwkJCXZsID0gU1ZFX1ZMX01JTjsKKworCQljdXJyZW50LT50 aHJlYWQuc3ZlX3ZsID0gdmw7CisJfQorCiAJc2V0X3RocmVhZF9mbGFnKFRJRl9GT1JFSUdOX0ZQ U1RBVEUpOwogCiAJbG9jYWxfYmhfZW5hYmxlKCk7CkBAIC0xODIsNiArNDkxLDkgQEAgdm9pZCBm cHNpbWRfZmx1c2hfdGhyZWFkKHZvaWQpCiAvKgogICogU2F2ZSB0aGUgdXNlcmxhbmQgRlBTSU1E IHN0YXRlIG9mICdjdXJyZW50JyB0byBtZW1vcnksIGJ1dCBvbmx5IGlmIHRoZSBzdGF0ZQogICog Y3VycmVudGx5IGhlbGQgaW4gdGhlIHJlZ2lzdGVycyBkb2VzIGluIGZhY3QgYmVsb25nIHRvICdj dXJyZW50JworICoKKyAqIEN1cnJlbnRseSwgU1ZFIHRhc2tzIGNhbid0IGV4aXN0LCBzbyBqdXN0 IFdBUk4gaW4gdGhhdCBjYXNlLgorICogU3Vic2VxdWVudCBwYXRjaGVzIHdpbGwgYWRkIGZ1bGwg U1ZFIHN1cHBvcnQgaGVyZS4KICAqLwogdm9pZCBmcHNpbWRfcHJlc2VydmVfY3VycmVudF9zdGF0 ZSh2b2lkKQogewpAQCAtMTkzLDYgKzUwNSw4IEBAIHZvaWQgZnBzaW1kX3ByZXNlcnZlX2N1cnJl bnRfc3RhdGUodm9pZCkKIAlpZiAoIXRlc3RfdGhyZWFkX2ZsYWcoVElGX0ZPUkVJR05fRlBTVEFU RSkpCiAJCWZwc2ltZF9zYXZlX3N0YXRlKCZjdXJyZW50LT50aHJlYWQuZnBzaW1kX3N0YXRlKTsK IAorCVdBUk5fT05fT05DRSh0ZXN0X2FuZF9jbGVhcl90aHJlYWRfZmxhZyhUSUZfU1ZFKSk7CisK IAlsb2NhbF9iaF9lbmFibGUoKTsKIH0KIApAQCAtMjExLDcgKzUyNSw3IEBAIHZvaWQgZnBzaW1k X3Jlc3RvcmVfY3VycmVudF9zdGF0ZSh2b2lkKQogCWlmICh0ZXN0X2FuZF9jbGVhcl90aHJlYWRf ZmxhZyhUSUZfRk9SRUlHTl9GUFNUQVRFKSkgewogCQlzdHJ1Y3QgZnBzaW1kX3N0YXRlICpzdCA9 ICZjdXJyZW50LT50aHJlYWQuZnBzaW1kX3N0YXRlOwogCi0JCWZwc2ltZF9sb2FkX3N0YXRlKHN0 KTsKKwkJdGFza19mcHNpbWRfbG9hZCgpOwogCQlfX3RoaXNfY3B1X3dyaXRlKGZwc2ltZF9sYXN0 X3N0YXRlLCBzdCk7CiAJCXN0LT5jcHUgPSBzbXBfcHJvY2Vzc29yX2lkKCk7CiAJfQpAQCAtMzgw LDggKzY5NCw4IEBAIHN0YXRpYyBpbnQgZnBzaW1kX2NwdV9wbV9ub3RpZmllcihzdHJ1Y3Qgbm90 aWZpZXJfYmxvY2sgKnNlbGYsCiB7CiAJc3dpdGNoIChjbWQpIHsKIAljYXNlIENQVV9QTV9FTlRF UjoKLQkJaWYgKGN1cnJlbnQtPm1tICYmICF0ZXN0X3RocmVhZF9mbGFnKFRJRl9GT1JFSUdOX0ZQ U1RBVEUpKQotCQkJZnBzaW1kX3NhdmVfc3RhdGUoJmN1cnJlbnQtPnRocmVhZC5mcHNpbWRfc3Rh dGUpOworCQlpZiAoY3VycmVudC0+bW0pCisJCQl0YXNrX2Zwc2ltZF9zYXZlKCk7CiAJCXRoaXNf Y3B1X3dyaXRlKGZwc2ltZF9sYXN0X3N0YXRlLCBOVUxMKTsKIAkJYnJlYWs7CiAJY2FzZSBDUFVf UE1fRVhJVDoKZGlmZiAtLWdpdCBhL2FyY2gvYXJtNjQva2VybmVsL3Byb2Nlc3MuYyBiL2FyY2gv YXJtNjQva2VybmVsL3Byb2Nlc3MuYwppbmRleCAyZGMwZjg0Li45YTNjNTYxIDEwMDY0NAotLS0g YS9hcmNoL2FybTY0L2tlcm5lbC9wcm9jZXNzLmMKKysrIGIvYXJjaC9hcm02NC9rZXJuZWwvcHJv Y2Vzcy5jCkBAIC00OSw2ICs0OSw3IEBACiAjaW5jbHVkZSA8bGludXgvbm90aWZpZXIuaD4KICNp bmNsdWRlIDx0cmFjZS9ldmVudHMvcG93ZXIuaD4KICNpbmNsdWRlIDxsaW51eC9wZXJjcHUuaD4K KyNpbmNsdWRlIDxsaW51eC90aHJlYWRfaW5mby5oPgogCiAjaW5jbHVkZSA8YXNtL2FsdGVybmF0 aXZlLmg+CiAjaW5jbHVkZSA8YXNtL2NvbXBhdC5oPgpAQCAtMjQxLDExICsyNDIsMjcgQEAgdm9p ZCByZWxlYXNlX3RocmVhZChzdHJ1Y3QgdGFza19zdHJ1Y3QgKmRlYWRfdGFzaykKIHsKIH0KIAor dm9pZCBhcmNoX3JlbGVhc2VfdGFza19zdHJ1Y3Qoc3RydWN0IHRhc2tfc3RydWN0ICp0c2spCit7 CisJZnBzaW1kX3JlbGVhc2VfdGFzayh0c2spOworfQorCisvKgorICogc3JjIGFuZCBkc3QgbWF5 IHRlbXBvcmFyaWx5IGhhdmUgYWxpYXNlZCBzdmVfc3RhdGUgYWZ0ZXIgdGFza19zdHJ1Y3QKKyAq IGlzIGNvcGllZC4gIFdlIGNhbm5vdCBmaXggdGhpcyBwcm9wZXJseSBoZXJlLCBiZWNhdXNlIHNy YyBtYXkgaGF2ZQorICogbGl2ZSBTVkUgc3RhdGUgYW5kIGRzdCdzIHRocmVhZF9pbmZvIG1heSBu b3QgZXhpc3QgeWV0LCBzbyB0d2Vha2luZworICogZWl0aGVyIHNyYydzIG9yIGRzdCdzIFRJRl9T VkUgaXMgbm90IHNhZmUuCisgKgorICogVGhlIHVuYWxpYXNpbmcgaXMgZG9uZSBpbiBjb3B5X3Ro cmVhZCgpIGluc3RlYWQuICBUaGlzIHdvcmtzIGJlY2F1c2UKKyAqIGRzdCBpcyBub3Qgc2NoZWR1 bGFibGUgb3IgdHJhY2VhYmxlIHVudGlsIGJvdGggb2YgdGhlc2UgZnVuY3Rpb25zCisgKiBoYXZl IGJlZW4gY2FsbGVkLgorICovCiBpbnQgYXJjaF9kdXBfdGFza19zdHJ1Y3Qoc3RydWN0IHRhc2tf c3RydWN0ICpkc3QsIHN0cnVjdCB0YXNrX3N0cnVjdCAqc3JjKQogewogCWlmIChjdXJyZW50LT5t bSkKIAkJZnBzaW1kX3ByZXNlcnZlX2N1cnJlbnRfc3RhdGUoKTsKIAkqZHN0ID0gKnNyYzsKKwog CXJldHVybiAwOwogfQogCkBAIC0yNTgsNiArMjc1LDEzIEBAIGludCBjb3B5X3RocmVhZCh1bnNp Z25lZCBsb25nIGNsb25lX2ZsYWdzLCB1bnNpZ25lZCBsb25nIHN0YWNrX3N0YXJ0LAogCiAJbWVt c2V0KCZwLT50aHJlYWQuY3B1X2NvbnRleHQsIDAsIHNpemVvZihzdHJ1Y3QgY3B1X2NvbnRleHQp KTsKIAorCS8qCisJICogVW5hbGlhcyBwLT50aHJlYWQuc3ZlX3N0YXRlIChpZiBhbnkpIGZyb20g dGhlIHBhcmVudCB0YXNrCisJICogYW5kIGRpc2FibGUgZGlzY2FyZCBTVkUgc3RhdGUgZm9yIHA6 CisJICovCisJY2xlYXJfdHNrX3RocmVhZF9mbGFnKHAsIFRJRl9TVkUpOworCXAtPnRocmVhZC5z dmVfc3RhdGUgPSBOVUxMOworCiAJaWYgKGxpa2VseSghKHAtPmZsYWdzICYgUEZfS1RIUkVBRCkp KSB7CiAJCSpjaGlsZHJlZ3MgPSAqY3VycmVudF9wdF9yZWdzKCk7CiAJCWNoaWxkcmVncy0+cmVn c1swXSA9IDA7CmRpZmYgLS1naXQgYS9hcmNoL2FybTY0L2tlcm5lbC90cmFwcy5jIGIvYXJjaC9h cm02NC9rZXJuZWwvdHJhcHMuYwppbmRleCAyMGZiZTQyLi43NzQxZGM5IDEwMDY0NAotLS0gYS9h cmNoL2FybTY0L2tlcm5lbC90cmFwcy5jCisrKyBiL2FyY2gvYXJtNjQva2VybmVsL3RyYXBzLmMK QEAgLTM2OSw4ICszNjksOCBAQCBzdGF0aWMgaW50IGNhbGxfdW5kZWZfaG9vayhzdHJ1Y3QgcHRf cmVncyAqcmVncykKIAlyZXR1cm4gZm4gPyBmbihyZWdzLCBpbnN0cikgOiAxOwogfQogCi1zdGF0 aWMgdm9pZCBmb3JjZV9zaWduYWxfaW5qZWN0KGludCBzaWduYWwsIGludCBjb2RlLCBzdHJ1Y3Qg cHRfcmVncyAqcmVncywKLQkJCQl1bnNpZ25lZCBsb25nIGFkZHJlc3MpCit2b2lkIGZvcmNlX3Np Z25hbF9pbmplY3QoaW50IHNpZ25hbCwgaW50IGNvZGUsIHN0cnVjdCBwdF9yZWdzICpyZWdzLAor CQkJIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKIHsKIAlzaWdpbmZvX3QgaW5mbzsKIAl2b2lkIF9f dXNlciAqcGMgPSAodm9pZCBfX3VzZXIgKilpbnN0cnVjdGlvbl9wb2ludGVyKHJlZ3MpOwpAQCAt Mzg0LDcgKzM4NCw3IEBAIHN0YXRpYyB2b2lkIGZvcmNlX3NpZ25hbF9pbmplY3QoaW50IHNpZ25h bCwgaW50IGNvZGUsIHN0cnVjdCBwdF9yZWdzICpyZWdzLAogCQlkZXNjID0gImlsbGVnYWwgbWVt b3J5IGFjY2VzcyI7CiAJCWJyZWFrOwogCWRlZmF1bHQ6Ci0JCWRlc2MgPSAiYmFkIG1vZGUiOwor CQlkZXNjID0gInVua25vd24gb3IgdW5yZWNvdmVyYWJsZSBlcnJvciI7CiAJCWJyZWFrOwogCX0K IAotLSAKMi4xLjQKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fCmt2bWFybSBtYWlsaW5nIGxpc3QKa3ZtYXJtQGxpc3RzLmNzLmNvbHVtYmlhLmVkdQpodHRw czovL2xpc3RzLmNzLmNvbHVtYmlhLmVkdS9tYWlsbWFuL2xpc3RpbmZvL2t2bWFybQo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from foss.arm.com ([217.140.101.70]:56834 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752427AbdJ0Kvl (ORCPT ); Fri, 27 Oct 2017 06:51:41 -0400 From: Dave Martin Subject: [PATCH v4 11/28] arm64/sve: Core task context handling Date: Fri, 27 Oct 2017 11:50:53 +0100 Message-ID: <1509101470-7881-12-git-send-email-Dave.Martin@arm.com> In-Reply-To: <1509101470-7881-1-git-send-email-Dave.Martin@arm.com> References: <1509101470-7881-1-git-send-email-Dave.Martin@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-arch-owner@vger.kernel.org List-ID: To: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas , Will Deacon , Ard Biesheuvel , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Szabolcs Nagy , Okamoto Takayuki , kvmarm@lists.cs.columbia.edu, libc-alpha@sourceware.org, linux-arch@vger.kernel.org Message-ID: <20171027105053.V2BRmRkIlj-I0-W2relthS3Jjc7KnP-kp5pt_fD5L4Y@z> This patch adds the core support for switching and managing the SVE architectural state of user tasks. Calls to the existing FPSIMD low-level save/restore functions are factored out as new functions task_fpsimd_{save,load}(), since SVE now dynamically may or may not need to be handled at these points depending on the kernel configuration, hardware features discovered at boot, and the runtime state of the task. To make these decisions as fast as possible, const cpucaps are used where feasible, via the system_supports_sve() helper. The SVE registers are only tracked for threads that have explicitly used SVE, indicated by the new thread flag TIF_SVE. Otherwise, the FPSIMD view of the architectural state is stored in thread.fpsimd_state as usual. When in use, the SVE registers are not stored directly in thread_struct due to their potentially large and variable size. Because the task_struct slab allocator must be configured very early during kernel boot, it is also tricky to configure it correctly to match the maximum vector length provided by the hardware, since this depends on examining secondary CPUs as well as the primary. Instead, a pointer sve_state in thread_struct points to a dynamically allocated buffer containing the SVE register data, and code is added to allocate and free this buffer at appropriate times. TIF_SVE is set when taking an SVE access trap from userspace, if suitable hardware support has been detected. This enables SVE for the thread: a subsequent return to userspace will disable the trap accordingly. If such a trap is taken without sufficient system- wide hardware support, SIGILL is sent to the thread instead as if an undefined instruction had been executed: this may happen if userspace tries to use SVE in a system where not all CPUs support it for example. The kernel will clear TIF_SVE and disable SVE for the thread whenever an explicit syscall is made by userspace. For backwards compatibility reasons and conformance with the spirit of the base AArch64 procedure call standard, the subset of the SVE register state that aliases the FPSIMD registers is still preserved across a syscall even if this happens. The remainder of the SVE register state logically becomes zero at syscall entry, though the actual zeroing work is currently deferred until the thread next tries to use SVE, causing another trap to the kernel. This implementation is suboptimal: in the future, the fastpath case may be optimised to zero the registers in-place and leave SVE enabled for the task, where beneficial. TIF_SVE is also cleared in the following slowpath cases, which are taken as reasonable hints that the task may no longer use SVE: * exec * fork and clone Code is added to sync data between thread.fpsimd_state and thread.sve_state whenever enabling/disabling SVE, in a manner consistent with the SVE architectural programmer's model. Signed-off-by: Dave Martin Cc: Ard Biesheuvel Cc: Alex BennĂ©e --- Changes since v3 ---------------- Changes requested by Catalin Marinas: * Change static __maybe_unused functions to static inlines. * Move fpsimd_release_thread() to the arch_release_task_struct() path. This avoids any need to worry about preemption. A declaration for arch_release_task_struct() is added to to help guard against inconsistent prototypes, though this is not required and the code would work without it. (thread_info.h is already included by the relevant code code, and arch/sh uses the same header). * Added comments to explain the meaning of TIF_SVE. * ABI change: Zero SVE regs on syscall entry. Detailed rationale: As currently documented, no guarantee is made about what a user task sees in the SVE registers after a syscall, except that V0-V31 (corresponding to Z0-Z31 bits [127:0]) are preserved. The actual kernel behaviour currently implemented is that the SVE registers are zeroed if a context switch or signal delivery occurs during a syscall. After a fork() or clone(), the SVE registers of the child task are zeroed also. The SVE registers are otherwise preserved. Flexibility is retained in the ABI about the exact criteria for the decision. There are some potential problems with this approach. Syscall impact -------------- Will, Catalin and Mark have expressed concerns about the risk of creating de facto ABI here: in scenarios or workloads where a context switch never occurs or is very unlikely, userspace may learn to rely on preservation of the SVE registers across certain syscalls. It is difficult to assess the impact of this: the syscall ABI is not a general-purpose interface, since it is usually hidden behind libc wrappers: direct invocation of SVC is discouraged. However, specialised runtimes, statically linked programs and binary blobs may bake in direct syscalls that make bad assumptions. Conversely, the relative cost of zeroing the SVE regs to mitigate against this also cannot be well characterised until SVE hardware exists. ptrace impact ------------- The current implementation can discard and zero the SVE registers at any point during a syscall, including before, after or between ptrace traps inside a single syscall. This means that setting the SVE registers through PTRACE_SETREGSET will often not do what the user expects: the new register values are only guaranteed to survive as far as userspace if set from an asynchronous signal-delivery-stop (e.g., breakpoint, SEGV or asynchronous signal delivered outside syscall context). This is consistent with the currently documented SVE user ABI, but likely to be surprising for a debugger user, since setting most registers of a tracee doesn't behave in this way. This patch ---------- The common syscall entry path is modified to forcibly discard SVE, and the discard logic elsewhere is removed. This means that there is a mandatory additional trap to the kernel when a user task tries to use SVE again after a syscall. This can be expensive for programs that use SVE heavily around syscalls, but can be optimised later. Because all ptrace traps occur after syscall entry, this also means that no discard will occur after SVE registers are set through ptrace, until the task next passes through userspace and does another syscall. The only exception to this is in fork()/clone() where forced discard still occurs: however, this is likely to be encountered less frequently by debugger users. This ABI change will commit the kernel to some extra cost: at least some thread flag manipulation, conditional branching and mov v0.16b, v0.16b mov v1.16b, v1.16b // ... mov v31.16b, v31.16b pfalse p0.b // ... pfalse p15.b wrffr p0.b (or equivalent) on the syscall fast path, for SVE tasks. For now this patch requires trap + memset() + reload, so is significantly more expensive. --- arch/arm64/include/asm/fpsimd.h | 16 ++ arch/arm64/include/asm/processor.h | 2 + arch/arm64/include/asm/thread_info.h | 4 + arch/arm64/include/asm/traps.h | 2 + arch/arm64/kernel/entry.S | 39 ++++- arch/arm64/kernel/fpsimd.c | 324 ++++++++++++++++++++++++++++++++++- arch/arm64/kernel/process.c | 24 +++ arch/arm64/kernel/traps.c | 6 +- 8 files changed, 406 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 026a7c7..5655fe1 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -20,6 +20,8 @@ #ifndef __ASSEMBLY__ +#include + /* * FP/SIMD storage area has: * - FPSR and FPCR @@ -72,6 +74,20 @@ extern void sve_load_state(void const *state, u32 const *pfpsr, unsigned long vq_minus_1); extern unsigned int sve_get_vl(void); +#ifdef CONFIG_ARM64_SVE + +extern size_t sve_state_size(struct task_struct const *task); + +extern void sve_alloc(struct task_struct *task); +extern void fpsimd_release_task(struct task_struct *task); + +#else /* ! CONFIG_ARM64_SVE */ + +static inline void sve_alloc(struct task_struct *task) { } +static inline void fpsimd_release_task(struct task_struct *task) { } + +#endif /* ! CONFIG_ARM64_SVE */ + /* For use by EFI runtime services calls only */ extern void __efi_fpsimd_begin(void); extern void __efi_fpsimd_end(void); diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 7dddca2..e2f575d 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -105,6 +105,8 @@ struct thread_struct { unsigned long tp2_value; #endif struct fpsimd_state fpsimd_state; + void *sve_state; /* SVE registers, if any */ + unsigned int sve_vl; /* SVE vector length */ unsigned long fault_address; /* fault info */ unsigned long fault_code; /* ESR_EL1 value */ struct debug_info debug; /* debugging */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index ddded64..92b7b48 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -63,6 +63,8 @@ struct thread_info { void arch_setup_new_exec(void); #define arch_setup_new_exec arch_setup_new_exec +void arch_release_task_struct(struct task_struct *tsk); + #endif /* @@ -92,6 +94,7 @@ void arch_setup_new_exec(void); #define TIF_RESTORE_SIGMASK 20 #define TIF_SINGLESTEP 21 #define TIF_32BIT 22 /* 32bit process */ +#define TIF_SVE 23 /* Scalable Vector Extension in use */ #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) @@ -105,6 +108,7 @@ void arch_setup_new_exec(void); #define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_FSCHECK (1 << TIF_FSCHECK) #define _TIF_32BIT (1 << TIF_32BIT) +#define _TIF_SVE (1 << TIF_SVE) #define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h index 45e3da3..1696f9d 100644 --- a/arch/arm64/include/asm/traps.h +++ b/arch/arm64/include/asm/traps.h @@ -34,6 +34,8 @@ struct undef_hook { void register_undef_hook(struct undef_hook *hook); void unregister_undef_hook(struct undef_hook *hook); +void force_signal_inject(int signal, int code, struct pt_regs *regs, + unsigned long address); void arm64_notify_segfault(struct pt_regs *regs, unsigned long addr); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index f5e851e..56e848f 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -607,6 +607,8 @@ el0_sync: b.eq el0_ia cmp x24, #ESR_ELx_EC_FP_ASIMD // FP/ASIMD access b.eq el0_fpsimd_acc + cmp x24, #ESR_ELx_EC_SVE // SVE access + b.eq el0_sve_acc cmp x24, #ESR_ELx_EC_FP_EXC64 // FP/ASIMD exception b.eq el0_fpsimd_exc cmp x24, #ESR_ELx_EC_SYS64 // configurable trap @@ -658,6 +660,7 @@ el0_svc_compat: /* * AArch32 syscall handling */ + ldr x16, [tsk, #TSK_TI_FLAGS] // load thread flags adrp stbl, compat_sys_call_table // load compat syscall table pointer mov wscno, w7 // syscall number in w7 (r7) mov wsc_nr, #__NR_compat_syscalls @@ -705,9 +708,19 @@ el0_fpsimd_acc: mov x1, sp bl do_fpsimd_acc b ret_to_user +el0_sve_acc: + /* + * Scalable Vector Extension access + */ + enable_dbg_and_irq + ct_user_exit + mov x0, x25 + mov x1, sp + bl do_sve_acc + b ret_to_user el0_fpsimd_exc: /* - * Floating Point or Advanced SIMD exception + * Floating Point, Advanced SIMD or SVE exception */ enable_dbg ct_user_exit @@ -835,16 +848,36 @@ ENDPROC(ret_to_user) */ .align 6 el0_svc: + ldr x16, [tsk, #TSK_TI_FLAGS] // load thread flags adrp stbl, sys_call_table // load syscall table pointer mov wscno, w8 // syscall number in w8 mov wsc_nr, #__NR_syscalls + +#ifndef CONFIG_ARM64_SVE + b el0_svc_naked +#else + tbz x16, #TIF_SVE, el0_svc_naked // Skip unless TIF_SVE set: + bic x16, x16, #_TIF_SVE // discard SVE state + str x16, [tsk, #TSK_TI_FLAGS] + + /* + * task_fpsimd_load() won't be called to update CPACR_EL1 in + * ret_to_user unless TIF_FOREIGN_FPSTATE is still set, which only + * happens if a context switch or kernel_neon_begin() or context + * modification (sigreturn, ptrace) intervenes. + * So, ensure that CPACR_EL1 is already correct for the fast-path case: + */ + mrs x9, cpacr_el1 + bic x9, x9, #CPACR_EL1_ZEN_EL0EN // disable SVE for el0 + msr cpacr_el1, x9 // synchronised by eret to el0 +#endif /* CONFIG_ARM64_SVE */ + el0_svc_naked: // compat entry point stp x0, xscno, [sp, #S_ORIG_X0] // save the original x0 and syscall number enable_dbg_and_irq ct_user_exit 1 - ldr x16, [tsk, #TSK_TI_FLAGS] // check for syscall hooks - tst x16, #_TIF_SYSCALL_WORK + tst x16, #_TIF_SYSCALL_WORK // check for syscall hooks b.ne __sys_trace cmp wscno, wsc_nr // check upper syscall limit b.hs ni_sys diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 7a865c8..0b15a594 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -18,18 +18,26 @@ */ #include +#include +#include #include #include #include +#include #include #include #include +#include #include #include +#include #include #include #include +#include +#include +#include #define FPEXC_IOF (1 << 0) #define FPEXC_DZF (1 << 1) @@ -39,6 +47,8 @@ #define FPEXC_IDF (1 << 7) /* + * (Note: in this discussion, statements about FPSIMD apply equally to SVE.) + * * In order to reduce the number of times the FPSIMD state is needlessly saved * and restored, we need to keep track of two things: * (a) for each task, we need to remember which CPU was the last one to have @@ -100,6 +110,279 @@ static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state); /* + * Call __sve_free() directly only if you know task can't be scheduled + * or preempted. + */ +static void __sve_free(struct task_struct *task) +{ + kfree(task->thread.sve_state); + task->thread.sve_state = NULL; +} + +static void sve_free(struct task_struct *task) +{ + WARN_ON(test_tsk_thread_flag(task, TIF_SVE)); + + __sve_free(task); +} + + +/* Offset of FFR in the SVE register dump */ +static size_t sve_ffr_offset(int vl) +{ + return SVE_SIG_FFR_OFFSET(sve_vq_from_vl(vl)) - SVE_SIG_REGS_OFFSET; +} + +static void *sve_pffr(struct task_struct *task) +{ + return (char *)task->thread.sve_state + + sve_ffr_offset(task->thread.sve_vl); +} + +static void change_cpacr(u64 val, u64 mask) +{ + u64 cpacr = read_sysreg(CPACR_EL1); + u64 new = (cpacr & ~mask) | val; + + if (new != cpacr) + write_sysreg(new, CPACR_EL1); +} + +static void sve_user_disable(void) +{ + change_cpacr(0, CPACR_EL1_ZEN_EL0EN); +} + +static void sve_user_enable(void) +{ + change_cpacr(CPACR_EL1_ZEN_EL0EN, CPACR_EL1_ZEN_EL0EN); +} + +/* + * TIF_SVE controls whether a task can use SVE without trapping while + * in userspace, and also the way a task's FPSIMD/SVE state is stored + * in thread_struct. + * + * The kernel uses this flag to track whether a user task is actively + * using SVE, and therefore whether full SVE register state needs to + * be tracked. If not, the cheaper FPSIMD context handling code can + * be used instead of the more costly SVE equivalents. + * + * * TIF_SVE set: + * + * The task can execute SVE instructions while in userspace without + * trapping to the kernel. + * + * When stored, Z0-Z31 (incorporating Vn in bits[127:0] or the + * corresponding Zn), P0-P15 and FFR are encoded in in + * task->thread.sve_state, formatted appropriately for vector + * length task->thread.sve_vl. + * + * task->thread.sve_state must point to a valid buffer at least + * sve_state_size(task) bytes in size. + * + * During any syscall, the kernel may optionally clear TIF_SVE and + * discard the vector state except for the FPSIMD subset. + * + * * TIF_SVE clear: + * + * An attempt by the user task to execute an SVE instruction causes + * do_sve_acc() to be called, which does some preparation and then + * sets TIF_SVE. + * + * When stored, FPSIMD registers V0-V31 are encoded in + * task->fpsimd_state; bits [max : 128] for each of Z0-Z31 are + * logically zero but not stored anywhere; P0-P15 and FFR are not + * stored and have unspecified values from userspace's point of + * view. For hygiene purposes, the kernel zeroes them on next use, + * but userspace is discouraged from relying on this. + * + * task->thread.sve_state does not need to be non-NULL, valid or any + * particular size: it must not be dereferenced. + * + * * FPSR and FPCR are always stored in task->fpsimd_state irrespctive of + * whether TIF_SVE is clear or set, since these are not vector length + * dependent. + */ + +/* + * Update current's FPSIMD/SVE registers from thread_struct. + * + * This function should be called only when the FPSIMD/SVE state in + * thread_struct is known to be up to date, when preparing to enter + * userspace. + * + * Softirqs (and preemption) must be disabled. + */ +static void task_fpsimd_load(void) +{ + WARN_ON(!in_softirq() && !irqs_disabled()); + + if (system_supports_sve() && test_thread_flag(TIF_SVE)) + sve_load_state(sve_pffr(current), + ¤t->thread.fpsimd_state.fpsr, + sve_vq_from_vl(current->thread.sve_vl) - 1); + else + fpsimd_load_state(¤t->thread.fpsimd_state); + + if (system_supports_sve()) { + /* Toggle SVE trapping for userspace if needed */ + if (test_thread_flag(TIF_SVE)) + sve_user_enable(); + else + sve_user_disable(); + + /* Serialised by exception return to user */ + } +} + +/* + * Ensure current's FPSIMD/SVE storage in thread_struct is up to date + * with respect to the CPU registers. + * + * Softirqs (and preemption) must be disabled. + */ +static void task_fpsimd_save(void) +{ + WARN_ON(!in_softirq() && !irqs_disabled()); + + if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { + if (system_supports_sve() && test_thread_flag(TIF_SVE)) { + if (WARN_ON(sve_get_vl() != current->thread.sve_vl)) { + /* + * Can't save the user regs, so current would + * re-enter user with corrupt state. + * There's no way to recover, so kill it: + */ + force_signal_inject( + SIGKILL, 0, current_pt_regs(), 0); + return; + } + + sve_save_state(sve_pffr(current), + ¤t->thread.fpsimd_state.fpsr); + } else + fpsimd_save_state(¤t->thread.fpsimd_state); + } +} + +#define ZREG(sve_state, vq, n) ((char *)(sve_state) + \ + (SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET)) + +/* + * Transfer the FPSIMD state in task->thread.fpsimd_state to + * task->thread.sve_state. + * + * Task can be a non-runnable task, or current. In the latter case, + * softirqs (and preemption) must be disabled. + * task->thread.sve_state must point to at least sve_state_size(task) + * bytes of allocated kernel memory. + * task->thread.fpsimd_state must be up to date before calling this function. + */ +static void fpsimd_to_sve(struct task_struct *task) +{ + unsigned int vq; + void *sst = task->thread.sve_state; + struct fpsimd_state const *fst = &task->thread.fpsimd_state; + unsigned int i; + + if (!system_supports_sve()) + return; + + vq = sve_vq_from_vl(task->thread.sve_vl); + for (i = 0; i < 32; ++i) + memcpy(ZREG(sst, vq, i), &fst->vregs[i], + sizeof(fst->vregs[i])); +} + +#ifdef CONFIG_ARM64_SVE + +/* + * Return how many bytes of memory are required to store the full SVE + * state for task, given task's currently configured vector length. + */ +size_t sve_state_size(struct task_struct const *task) +{ + return SVE_SIG_REGS_SIZE(sve_vq_from_vl(task->thread.sve_vl)); +} + +/* + * Ensure that task->thread.sve_state is allocated and sufficiently large. + * + * This function should be used only in preparation for replacing + * task->thread.sve_state with new data. The memory is always zeroed + * here to prevent stale data from showing through: this is done in + * the interest of testability and predictability: except in the + * do_sve_acc() case, there is no ABI requirement to hide stale data + * written previously be task. + */ +void sve_alloc(struct task_struct *task) +{ + if (task->thread.sve_state) { + memset(task->thread.sve_state, 0, sve_state_size(current)); + return; + } + + /* This is a small allocation (maximum ~8KB) and Should Not Fail. */ + task->thread.sve_state = + kzalloc(sve_state_size(task), GFP_KERNEL); + + /* + * If future SVE revisions can have larger vectors though, + * this may cease to be true: + */ + BUG_ON(!task->thread.sve_state); +} + +/* + * Called from the put_task_struct() path, which cannot get here + * unless dead_task is really dead and not schedulable. + */ +void fpsimd_release_task(struct task_struct *dead_task) +{ + __sve_free(dead_task); +} + +#endif /* CONFIG_ARM64_SVE */ + +/* + * Trapped SVE access + * + * Storage is allocated for the full SVE state, the current FPSIMD + * register contents are migrated across, and TIF_SVE is set so that + * the SVE access trap will be disabled the next time this task + * reaches ret_to_user. + * + * TIF_SVE should be clear on entry: otherwise, task_fpsimd_load() + * would have disabled the SVE access trap for userspace during + * ret_to_user, making an SVE access trap impossible in that case. + */ +void do_sve_acc(unsigned int esr, struct pt_regs *regs) +{ + /* Even if we chose not to use SVE, the hardware could still trap: */ + if (unlikely(!system_supports_sve()) || WARN_ON(is_compat_task())) { + force_signal_inject(SIGILL, ILL_ILLOPC, regs, 0); + return; + } + + sve_alloc(current); + + local_bh_disable(); + + task_fpsimd_save(); + fpsimd_to_sve(current); + + /* Force ret_to_user to reload the registers: */ + fpsimd_flush_task_state(current); + set_thread_flag(TIF_FOREIGN_FPSTATE); + + if (test_and_set_thread_flag(TIF_SVE)) + WARN_ON(1); /* SVE access shouldn't have trapped */ + + local_bh_enable(); +} + +/* * Trapped FP/ASIMD access. */ void do_fpsimd_acc(unsigned int esr, struct pt_regs *regs) @@ -144,8 +427,8 @@ void fpsimd_thread_switch(struct task_struct *next) * the registers is in fact the most recent userland FPSIMD state of * 'current'. */ - if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE)) - fpsimd_save_state(¤t->thread.fpsimd_state); + if (current->mm) + task_fpsimd_save(); if (next->mm) { /* @@ -167,6 +450,8 @@ void fpsimd_thread_switch(struct task_struct *next) void fpsimd_flush_thread(void) { + int vl; + if (!system_supports_fpsimd()) return; @@ -174,6 +459,30 @@ void fpsimd_flush_thread(void) memset(¤t->thread.fpsimd_state, 0, sizeof(struct fpsimd_state)); fpsimd_flush_task_state(current); + + if (system_supports_sve()) { + clear_thread_flag(TIF_SVE); + sve_free(current); + + /* + * Reset the task vector length as required. + * This is where we ensure that all user tasks have a valid + * vector length configured: no kernel task can become a user + * task without an exec and hence a call to this function. + * If a bug causes this to go wrong, we make some noise and + * try to fudge thread.sve_vl to a safe value here. + */ + vl = current->thread.sve_vl; + + if (vl == 0) + vl = SVE_VL_MIN; + + if (WARN_ON(!sve_vl_valid(vl))) + vl = SVE_VL_MIN; + + current->thread.sve_vl = vl; + } + set_thread_flag(TIF_FOREIGN_FPSTATE); local_bh_enable(); @@ -182,6 +491,9 @@ void fpsimd_flush_thread(void) /* * Save the userland FPSIMD state of 'current' to memory, but only if the state * currently held in the registers does in fact belong to 'current' + * + * Currently, SVE tasks can't exist, so just WARN in that case. + * Subsequent patches will add full SVE support here. */ void fpsimd_preserve_current_state(void) { @@ -193,6 +505,8 @@ void fpsimd_preserve_current_state(void) if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) fpsimd_save_state(¤t->thread.fpsimd_state); + WARN_ON_ONCE(test_and_clear_thread_flag(TIF_SVE)); + local_bh_enable(); } @@ -211,7 +525,7 @@ void fpsimd_restore_current_state(void) if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) { struct fpsimd_state *st = ¤t->thread.fpsimd_state; - fpsimd_load_state(st); + task_fpsimd_load(); __this_cpu_write(fpsimd_last_state, st); st->cpu = smp_processor_id(); } @@ -380,8 +694,8 @@ static int fpsimd_cpu_pm_notifier(struct notifier_block *self, { switch (cmd) { case CPU_PM_ENTER: - if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE)) - fpsimd_save_state(¤t->thread.fpsimd_state); + if (current->mm) + task_fpsimd_save(); this_cpu_write(fpsimd_last_state, NULL); break; case CPU_PM_EXIT: diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 2dc0f84..9a3c561 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -49,6 +49,7 @@ #include #include #include +#include #include #include @@ -241,11 +242,27 @@ void release_thread(struct task_struct *dead_task) { } +void arch_release_task_struct(struct task_struct *tsk) +{ + fpsimd_release_task(tsk); +} + +/* + * src and dst may temporarily have aliased sve_state after task_struct + * is copied. We cannot fix this properly here, because src may have + * live SVE state and dst's thread_info may not exist yet, so tweaking + * either src's or dst's TIF_SVE is not safe. + * + * The unaliasing is done in copy_thread() instead. This works because + * dst is not schedulable or traceable until both of these functions + * have been called. + */ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) { if (current->mm) fpsimd_preserve_current_state(); *dst = *src; + return 0; } @@ -258,6 +275,13 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start, memset(&p->thread.cpu_context, 0, sizeof(struct cpu_context)); + /* + * Unalias p->thread.sve_state (if any) from the parent task + * and disable discard SVE state for p: + */ + clear_tsk_thread_flag(p, TIF_SVE); + p->thread.sve_state = NULL; + if (likely(!(p->flags & PF_KTHREAD))) { *childregs = *current_pt_regs(); childregs->regs[0] = 0; diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 20fbe42..7741dc9 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -369,8 +369,8 @@ static int call_undef_hook(struct pt_regs *regs) return fn ? fn(regs, instr) : 1; } -static void force_signal_inject(int signal, int code, struct pt_regs *regs, - unsigned long address) +void force_signal_inject(int signal, int code, struct pt_regs *regs, + unsigned long address) { siginfo_t info; void __user *pc = (void __user *)instruction_pointer(regs); @@ -384,7 +384,7 @@ static void force_signal_inject(int signal, int code, struct pt_regs *regs, desc = "illegal memory access"; break; default: - desc = "bad mode"; + desc = "unknown or unrecoverable error"; break; } -- 2.1.4