From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Martin Subject: [PATCH v4 18/28] arm64/sve: Preserve SVE registers around EFI runtime service calls Date: Fri, 27 Oct 2017 11:51:00 +0100 Message-ID: <1509101470-7881-19-git-send-email-Dave.Martin@arm.com> References: <1509101470-7881-1-git-send-email-Dave.Martin@arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <1509101470-7881-1-git-send-email-Dave.Martin@arm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu To: linux-arm-kernel@lists.infradead.org Cc: linux-arch@vger.kernel.org, Okamoto Takayuki , libc-alpha@sourceware.org, Ard Biesheuvel , Szabolcs Nagy , Catalin Marinas , Will Deacon , kvmarm@lists.cs.columbia.edu List-Id: linux-arch.vger.kernel.org VGhlIEVGSSBydW50aW1lIHNlcnZpY2VzIEFCSSBhbGxvd3MgRUZJIHRvIG1ha2UgZnJlZSB1c2Ug b2YgdGhlCkZQU0lNRCByZWdpc3RlcnMgZHVyaW5nIEVGSSBydW50aW1lIHNlcnZpY2UgY2FsbHMs IHN1YmplY3QgdG8gdGhlCmNhbGxlZS1zYXZlIHJlcXVpcmVtZW50cyBvZiB0aGUgQUFyY2g2NCBw cm9jZWR1cmUgY2FsbCBzdGFuZGFyZC4KCkhvd2V2ZXIsIHRoZSBTVkUgYXJjaGl0ZWN0dXJlIGFs bG93cyB1cHBlciBiaXRzIG9mIHRoZSBTVkUgdmVjdG9yCnJlZ2lzdGVycyB0byBiZSB6ZXJvZWQg YXMgYSBzaWRlLWVmZmVjdCBvZiBGUFNJTUQgVi1yZWdpc3Rlcgp3cml0ZXMuICBUaGlzIG1lYW5z IHRoYXQgdGhlIFNWRSB2ZWN0b3IgcmVnaXN0ZXJzIG11c3QgYmUgc2F2ZWQgaW4KdGhlaXIgZW50 aXJldHkgaW4gb3JkZXIgdG8gYXZvaWQgZGF0YSBsb3NzOiBub24tU1ZFLWF3YXJlIEVGSQppbXBs ZW1lbnRhdGlvbnMgY2Fubm90IHJlc3RvcmUgdGhlbSBjb3JyZWN0bHkuCgpUaGUgbm9uLUlSUSBj YXNlIGlzIGFscmVhZHkgaGFuZGxlZCBncmFjZWZ1bGx5IGJ5Cmtlcm5lbF9uZW9uX2JlZ2luKCku ICBGb3IgdGhlIElSUSBjYXNlLCB0aGlzIHBhdGNoIGFsbG9jYXRlcyBhCnN1aXRhYmxlIHBlci1D UFUgc3Rhc2ggYnVmZmVyIGZvciB0aGUgZnVsbCBTVkUgcmVnaXN0ZXIgc3RhdGUgYW5kCnVzZXMg aXQgdG8gcHJlc2VydmUgdGhlIGFmZmVjdGVkIHJlZ2lzdGVycyBhcm91bmQgRUZJIGNhbGxzLiAg SXQgaXMKY3VycmVudGx5IHVuY2xlYXIgaG93IHRoZSBFRkkgcnVudGltZSBzZXJ2aWNlcyBBQkkg d2lsbCBiZQpjbGFyaWZpZWQgd2l0aCByZXNwZWN0IHRvIFNWRSwgc28gaXQgc2FmZXN0IHRvIGFz c3VtZSB0aGF0IHRoZQpwcmVkaWNhdGUgcmVnaXN0ZXJzIGFuZCBGRlIgbXVzdCBiZSBzYXZlZCBh bmQgcmVzdG9yZWQgdG9vLgoKTm8gYXR0ZW1wdCBpcyBtYWRlIHRvIHJlc3RvcmUgdGhlIHJlc3Rv cmUgdGhlIHZlY3RvciBsZW5ndGggYWZ0ZXIKYSBjYWxsLCBmb3Igbm93LiAgSXQgaXMgZGVlbWVk IHJhdGhlciBpbnNhbmUgZm9yIEVGSSB0byBjaGFuZ2UgaXQsCmFuZCBjb250ZW1wb3JhcnkgRUZJ IGltcGxlbWVudGF0aW9ucyBjZXJ0YWlubHkgd29uJ3QuCgpTaWduZWQtb2ZmLWJ5OiBEYXZlIE1h cnRpbiA8RGF2ZS5NYXJ0aW5AYXJtLmNvbT4KUmV2aWV3ZWQtYnk6IEFsZXggQmVubsOpZSA8YWxl eC5iZW5uZWVAbGluYXJvLm9yZz4KUmV2aWV3ZWQtYnk6IENhdGFsaW4gTWFyaW5hcyA8Y2F0YWxp bi5tYXJpbmFzQGFybS5jb20+CkNjOiBBcmQgQmllc2hldXZlbCA8YXJkLmJpZXNoZXV2ZWxAbGlu YXJvLm9yZz4KLS0tCiBhcmNoL2FybTY0L2tlcm5lbC9mcHNpbWQuYyB8IDY3ICsrKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrLS0tLS0KIDEgZmlsZSBjaGFuZ2VkLCA2MSBp bnNlcnRpb25zKCspLCA2IGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL2FyY2gvYXJtNjQva2Vy bmVsL2Zwc2ltZC5jIGIvYXJjaC9hcm02NC9rZXJuZWwvZnBzaW1kLmMKaW5kZXggZDUxMDhlOC4u ZWRjMjdkMiAxMDA2NDQKLS0tIGEvYXJjaC9hcm02NC9rZXJuZWwvZnBzaW1kLmMKKysrIGIvYXJj aC9hcm02NC9rZXJuZWwvZnBzaW1kLmMKQEAgLTEyMSwxMSArMTIxLDEzIEBAIHN0YXRpYyBpbnQg c3ZlX2RlZmF1bHRfdmwgPSAtMTsKIGludCBfX3JvX2FmdGVyX2luaXQgc3ZlX21heF92bCA9IC0x OwogLyogU2V0IG9mIGF2YWlsYWJsZSB2ZWN0b3IgbGVuZ3RocywgYXMgdnFfdG9fYml0KHZxKTog Ki8KIHN0YXRpYyBfX3JvX2FmdGVyX2luaXQgREVDTEFSRV9CSVRNQVAoc3ZlX3ZxX21hcCwgU1ZF X1ZRX01BWCk7CitzdGF0aWMgdm9pZCBfX3BlcmNwdSAqZWZpX3N2ZV9zdGF0ZTsKIAogI2Vsc2Ug LyogISBDT05GSUdfQVJNNjRfU1ZFICovCiAKIC8qIER1bW15IGRlY2xhcmF0aW9uIGZvciBjb2Rl IHRoYXQgd2lsbCBiZSBvcHRpbWlzZWQgb3V0OiAqLwogZXh0ZXJuIF9fcm9fYWZ0ZXJfaW5pdCBE RUNMQVJFX0JJVE1BUChzdmVfdnFfbWFwLCBTVkVfVlFfTUFYKTsKK2V4dGVybiB2b2lkIF9fcGVy Y3B1ICplZmlfc3ZlX3N0YXRlOwogCiAjZW5kaWYgLyogISBDT05GSUdfQVJNNjRfU1ZFICovCiAK QEAgLTU1MCw2ICs1NTIsMzAgQEAgaW50IHN2ZV92ZXJpZnlfdnFfbWFwKHZvaWQpCiAJcmV0dXJu IHJldDsKIH0KIAorc3RhdGljIHZvaWQgX19pbml0IHN2ZV9lZmlfc2V0dXAodm9pZCkKK3sKKwlp ZiAoIUlTX0VOQUJMRUQoQ09ORklHX0VGSSkpCisJCXJldHVybjsKKworCS8qCisJICogYWxsb2Nf cGVyY3B1KCkgd2FybnMgYW5kIHByaW50cyBhIGJhY2t0cmFjZSBpZiB0aGlzIGdvZXMgd3Jvbmcu CisJICogVGhpcyBpcyBldmlkZW5jZSBvZiBhIGNyaXBwbGVkIHN5c3RlbSBhbmQgd2UgYXJlIHJl dHVybmluZyB2b2lkLAorCSAqIHNvIG5vIGF0dGVtcHQgaXMgbWFkZSB0byBoYW5kbGUgdGhpcyBz aXR1YXRpb24gaGVyZS4KKwkgKi8KKwlpZiAoIXN2ZV92bF92YWxpZChzdmVfbWF4X3ZsKSkKKwkJ Z290byBmYWlsOworCisJZWZpX3N2ZV9zdGF0ZSA9IF9fYWxsb2NfcGVyY3B1KAorCQlTVkVfU0lH X1JFR1NfU0laRShzdmVfdnFfZnJvbV92bChzdmVfbWF4X3ZsKSksIFNWRV9WUV9CWVRFUyk7CisJ aWYgKCFlZmlfc3ZlX3N0YXRlKQorCQlnb3RvIGZhaWw7CisKKwlyZXR1cm47CisKK2ZhaWw6CisJ cGFuaWMoIkNhbm5vdCBhbGxvY2F0ZSBwZXJjcHUgbWVtb3J5IGZvciBFRkkgU1ZFIHNhdmUvcmVz dG9yZSIpOworfQorCiAvKgogICogRW5hYmxlIFNWRSBmb3IgRUwxLgogICogSW50ZW5kZWQgZm9y IHVzZSBieSB0aGUgY3B1ZmVhdHVyZXMgY29kZSBkdXJpbmcgQ1BVIGJvb3QuCkBAIC01OTcsNiAr NjIzLDggQEAgdm9pZCBfX2luaXQgc3ZlX3NldHVwKHZvaWQpCiAJCXN2ZV9tYXhfdmwpOwogCXBy X2luZm8oIlNWRTogZGVmYXVsdCB2ZWN0b3IgbGVuZ3RoICV1IGJ5dGVzIHBlciB2ZWN0b3JcbiIs CiAJCXN2ZV9kZWZhdWx0X3ZsKTsKKworCXN2ZV9lZmlfc2V0dXAoKTsKIH0KIAogLyoKQEAgLTky NSw2ICs5NTMsNyBAQCBFWFBPUlRfU1lNQk9MKGtlcm5lbF9uZW9uX2VuZCk7CiAKIHN0YXRpYyBE RUZJTkVfUEVSX0NQVShzdHJ1Y3QgZnBzaW1kX3N0YXRlLCBlZmlfZnBzaW1kX3N0YXRlKTsKIHN0 YXRpYyBERUZJTkVfUEVSX0NQVShib29sLCBlZmlfZnBzaW1kX3N0YXRlX3VzZWQpOworc3RhdGlj IERFRklORV9QRVJfQ1BVKGJvb2wsIGVmaV9zdmVfc3RhdGVfdXNlZCk7CiAKIC8qCiAgKiBFRkkg cnVudGltZSBzZXJ2aWNlcyBzdXBwb3J0IGZ1bmN0aW9ucwpAQCAtOTUwLDEwICs5NzksMjQgQEAg dm9pZCBfX2VmaV9mcHNpbWRfYmVnaW4odm9pZCkKIAogCVdBUk5fT04ocHJlZW1wdGlibGUoKSk7 CiAKLQlpZiAobWF5X3VzZV9zaW1kKCkpCisJaWYgKG1heV91c2Vfc2ltZCgpKSB7CiAJCWtlcm5l bF9uZW9uX2JlZ2luKCk7Ci0JZWxzZSB7Ci0JCWZwc2ltZF9zYXZlX3N0YXRlKHRoaXNfY3B1X3B0 cigmZWZpX2Zwc2ltZF9zdGF0ZSkpOworCX0gZWxzZSB7CisJCS8qCisJCSAqIElmICFlZmlfc3Zl X3N0YXRlLCBTVkUgY2FuJ3QgYmUgaW4gdXNlIHlldCBhbmQgZG9lc24ndCBuZWVkCisJCSAqIHBy ZXNlcnZpbmc6CisJCSAqLworCQlpZiAoc3lzdGVtX3N1cHBvcnRzX3N2ZSgpICYmIGxpa2VseShl Zmlfc3ZlX3N0YXRlKSkgeworCQkJY2hhciAqc3ZlX3N0YXRlID0gdGhpc19jcHVfcHRyKGVmaV9z dmVfc3RhdGUpOworCisJCQlfX3RoaXNfY3B1X3dyaXRlKGVmaV9zdmVfc3RhdGVfdXNlZCwgdHJ1 ZSk7CisKKwkJCXN2ZV9zYXZlX3N0YXRlKHN2ZV9zdGF0ZSArIHN2ZV9mZnJfb2Zmc2V0KHN2ZV9t YXhfdmwpLAorCQkJCSAgICAgICAmdGhpc19jcHVfcHRyKCZlZmlfZnBzaW1kX3N0YXRlKS0+ZnBz cik7CisJCX0gZWxzZSB7CisJCQlmcHNpbWRfc2F2ZV9zdGF0ZSh0aGlzX2NwdV9wdHIoJmVmaV9m cHNpbWRfc3RhdGUpKTsKKwkJfQorCiAJCV9fdGhpc19jcHVfd3JpdGUoZWZpX2Zwc2ltZF9zdGF0 ZV91c2VkLCB0cnVlKTsKIAl9CiB9CkBAIC05NjYsMTAgKzEwMDksMjIgQEAgdm9pZCBfX2VmaV9m cHNpbWRfZW5kKHZvaWQpCiAJaWYgKCFzeXN0ZW1fc3VwcG9ydHNfZnBzaW1kKCkpCiAJCXJldHVy bjsKIAotCWlmIChfX3RoaXNfY3B1X3hjaGcoZWZpX2Zwc2ltZF9zdGF0ZV91c2VkLCBmYWxzZSkp Ci0JCWZwc2ltZF9sb2FkX3N0YXRlKHRoaXNfY3B1X3B0cigmZWZpX2Zwc2ltZF9zdGF0ZSkpOwot CWVsc2UKKwlpZiAoIV9fdGhpc19jcHVfeGNoZyhlZmlfZnBzaW1kX3N0YXRlX3VzZWQsIGZhbHNl KSkgewogCQlrZXJuZWxfbmVvbl9lbmQoKTsKKwl9IGVsc2UgeworCQlpZiAoc3lzdGVtX3N1cHBv cnRzX3N2ZSgpICYmCisJCSAgICBsaWtlbHkoX190aGlzX2NwdV9yZWFkKGVmaV9zdmVfc3RhdGVf dXNlZCkpKSB7CisJCQljaGFyIGNvbnN0ICpzdmVfc3RhdGUgPSB0aGlzX2NwdV9wdHIoZWZpX3N2 ZV9zdGF0ZSk7CisKKwkJCXN2ZV9sb2FkX3N0YXRlKHN2ZV9zdGF0ZSArIHN2ZV9mZnJfb2Zmc2V0 KHN2ZV9tYXhfdmwpLAorCQkJCSAgICAgICAmdGhpc19jcHVfcHRyKCZlZmlfZnBzaW1kX3N0YXRl KS0+ZnBzciwKKwkJCQkgICAgICAgc3ZlX3ZxX2Zyb21fdmwoc3ZlX2dldF92bCgpKSAtIDEpOwor CisJCQlfX3RoaXNfY3B1X3dyaXRlKGVmaV9zdmVfc3RhdGVfdXNlZCwgZmFsc2UpOworCQl9IGVs c2UgeworCQkJZnBzaW1kX2xvYWRfc3RhdGUodGhpc19jcHVfcHRyKCZlZmlfZnBzaW1kX3N0YXRl KSk7CisJCX0KKwl9CiB9CiAKICNlbmRpZiAvKiBDT05GSUdfRUZJICovCi0tIAoyLjEuNAoKX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18Ka3ZtYXJtIG1haWxp bmcgbGlzdAprdm1hcm1AbGlzdHMuY3MuY29sdW1iaWEuZWR1Cmh0dHBzOi8vbGlzdHMuY3MuY29s dW1iaWEuZWR1L21haWxtYW4vbGlzdGluZm8va3ZtYXJtCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:56966 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752496AbdJ0Kvz (ORCPT ); Fri, 27 Oct 2017 06:51:55 -0400 From: Dave Martin Subject: [PATCH v4 18/28] arm64/sve: Preserve SVE registers around EFI runtime service calls Date: Fri, 27 Oct 2017 11:51:00 +0100 Message-ID: <1509101470-7881-19-git-send-email-Dave.Martin@arm.com> In-Reply-To: <1509101470-7881-1-git-send-email-Dave.Martin@arm.com> References: <1509101470-7881-1-git-send-email-Dave.Martin@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-arch-owner@vger.kernel.org List-ID: To: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas , Will Deacon , Ard Biesheuvel , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Szabolcs Nagy , Okamoto Takayuki , kvmarm@lists.cs.columbia.edu, libc-alpha@sourceware.org, linux-arch@vger.kernel.org Message-ID: <20171027105100.52oKrlHwEIIuOmfWBSZORQzj8fIgWTiWfV8aU4easZY@z> The EFI runtime services ABI allows EFI to make free use of the FPSIMD registers during EFI runtime service calls, subject to the callee-save requirements of the AArch64 procedure call standard. However, the SVE architecture allows upper bits of the SVE vector registers to be zeroed as a side-effect of FPSIMD V-register writes. This means that the SVE vector registers must be saved in their entirety in order to avoid data loss: non-SVE-aware EFI implementations cannot restore them correctly. The non-IRQ case is already handled gracefully by kernel_neon_begin(). For the IRQ case, this patch allocates a suitable per-CPU stash buffer for the full SVE register state and uses it to preserve the affected registers around EFI calls. It is currently unclear how the EFI runtime services ABI will be clarified with respect to SVE, so it safest to assume that the predicate registers and FFR must be saved and restored too. No attempt is made to restore the restore the vector length after a call, for now. It is deemed rather insane for EFI to change it, and contemporary EFI implementations certainly won't. Signed-off-by: Dave Martin Reviewed-by: Alex Bennée Reviewed-by: Catalin Marinas Cc: Ard Biesheuvel --- arch/arm64/kernel/fpsimd.c | 67 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 61 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index d5108e8..edc27d2 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -121,11 +121,13 @@ static int sve_default_vl = -1; int __ro_after_init sve_max_vl = -1; /* Set of available vector lengths, as vq_to_bit(vq): */ static __ro_after_init DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); +static void __percpu *efi_sve_state; #else /* ! CONFIG_ARM64_SVE */ /* Dummy declaration for code that will be optimised out: */ extern __ro_after_init DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); +extern void __percpu *efi_sve_state; #endif /* ! CONFIG_ARM64_SVE */ @@ -550,6 +552,30 @@ int sve_verify_vq_map(void) return ret; } +static void __init sve_efi_setup(void) +{ + if (!IS_ENABLED(CONFIG_EFI)) + return; + + /* + * alloc_percpu() warns and prints a backtrace if this goes wrong. + * This is evidence of a crippled system and we are returning void, + * so no attempt is made to handle this situation here. + */ + if (!sve_vl_valid(sve_max_vl)) + goto fail; + + efi_sve_state = __alloc_percpu( + SVE_SIG_REGS_SIZE(sve_vq_from_vl(sve_max_vl)), SVE_VQ_BYTES); + if (!efi_sve_state) + goto fail; + + return; + +fail: + panic("Cannot allocate percpu memory for EFI SVE save/restore"); +} + /* * Enable SVE for EL1. * Intended for use by the cpufeatures code during CPU boot. @@ -597,6 +623,8 @@ void __init sve_setup(void) sve_max_vl); pr_info("SVE: default vector length %u bytes per vector\n", sve_default_vl); + + sve_efi_setup(); } /* @@ -925,6 +953,7 @@ EXPORT_SYMBOL(kernel_neon_end); static DEFINE_PER_CPU(struct fpsimd_state, efi_fpsimd_state); static DEFINE_PER_CPU(bool, efi_fpsimd_state_used); +static DEFINE_PER_CPU(bool, efi_sve_state_used); /* * EFI runtime services support functions @@ -950,10 +979,24 @@ void __efi_fpsimd_begin(void) WARN_ON(preemptible()); - if (may_use_simd()) + if (may_use_simd()) { kernel_neon_begin(); - else { - fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state)); + } else { + /* + * If !efi_sve_state, SVE can't be in use yet and doesn't need + * preserving: + */ + if (system_supports_sve() && likely(efi_sve_state)) { + char *sve_state = this_cpu_ptr(efi_sve_state); + + __this_cpu_write(efi_sve_state_used, true); + + sve_save_state(sve_state + sve_ffr_offset(sve_max_vl), + &this_cpu_ptr(&efi_fpsimd_state)->fpsr); + } else { + fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state)); + } + __this_cpu_write(efi_fpsimd_state_used, true); } } @@ -966,10 +1009,22 @@ void __efi_fpsimd_end(void) if (!system_supports_fpsimd()) return; - if (__this_cpu_xchg(efi_fpsimd_state_used, false)) - fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state)); - else + if (!__this_cpu_xchg(efi_fpsimd_state_used, false)) { kernel_neon_end(); + } else { + if (system_supports_sve() && + likely(__this_cpu_read(efi_sve_state_used))) { + char const *sve_state = this_cpu_ptr(efi_sve_state); + + sve_load_state(sve_state + sve_ffr_offset(sve_max_vl), + &this_cpu_ptr(&efi_fpsimd_state)->fpsr, + sve_vq_from_vl(sve_get_vl()) - 1); + + __this_cpu_write(efi_sve_state_used, false); + } else { + fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state)); + } + } } #endif /* CONFIG_EFI */ -- 2.1.4