From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Martin Subject: [PATCH v2 18/28] arm64/sve: Preserve SVE registers around EFI runtime service calls Date: Thu, 31 Aug 2017 18:00:50 +0100 Message-ID: <1504198860-12951-19-git-send-email-Dave.Martin@arm.com> References: <1504198860-12951-1-git-send-email-Dave.Martin@arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <1504198860-12951-1-git-send-email-Dave.Martin@arm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu To: linux-arm-kernel@lists.infradead.org Cc: linux-arch@vger.kernel.org, libc-alpha@sourceware.org, Ard Biesheuvel , Szabolcs Nagy , Catalin Marinas , Will Deacon , Richard Sandiford , kvmarm@lists.cs.columbia.edu List-Id: linux-arch.vger.kernel.org VGhlIEVGSSBydW50aW1lIHNlcnZpY2VzIEFCSSBhbGxvd3MgRUZJIHRvIG1ha2UgZnJlZSB1c2Ug b2YgdGhlCkZQU0lNRCByZWdpc3RlcnMgZHVyaW5nIEVGSSBydW50aW1lIHNlcnZpY2UgY2FsbHMs IHN1YmplY3QgdG8gdGhlCmNhbGxlZS1zYXZlIHJlcXVpcmVtZW50cyBvZiB0aGUgQUFyY2g2NCBw cm9jZWR1cmUgY2FsbCBzdGFuZGFyZC4KCkhvd2V2ZXIsIHRoZSBTVkUgYXJjaGl0ZWN0dXJlIGFs bG93cyB1cHBlciBiaXRzIG9mIHRoZSBTVkUgdmVjdG9yCnJlZ2lzdGVycyB0byBiZSB6ZXJvZWQg YXMgYSBzaWRlLWVmZmVjdCBvZiBGUFNJTUQgVi1yZWdpc3Rlcgp3cml0ZXMuICBUaGlzIG1lYW5z IHRoYXQgdGhlIFNWRSB2ZWN0b3IgcmVnaXN0ZXJzIG11c3QgYmUgc2F2ZWQgaW4KdGhlaXIgZW50 aXJldHkgaW4gb3JkZXIgdG8gYXZvaWQgZGF0YSBsb3NzOiBub24tU1ZFLWF3YXJlIEVGSQppbXBs ZW1lbnRhdGlvbnMgY2Fubm90IHJlc3RvcmUgdGhlbSBjb3JyZWN0bHkuCgpUaGUgbm9uLUlSUSBj YXNlIGlzIGFscmVhZHkgaGFuZGxlZCBncmFjZWZ1bGx5IGJ5Cmtlcm5lbF9uZW9uX2JlZ2luKCku ICBGb3IgdGhlIElSUSBjYXNlLCB0aGlzIHBhdGNoIGFsbG9jYXRlcyBhCnN1aXRhYmxlIHBlci1D UFUgc3Rhc2ggYnVmZmVyIGZvciB0aGUgZnVsbCBTVkUgcmVnaXN0ZXIgc3RhdGUgYW5kCnVzZXMg aXQgdG8gcHJlc2VydmUgdGhlIGFmZmVjdGVkIHJlZ2lzdGVycyBhcm91bmQgRUZJIGNhbGxzLiAg SXQgaXMKY3VycmVudGx5IHVuY2xlYXIgaG93IHRoZSBFRkkgcnVudGltZSBzZXJ2aWNlcyBBQkkg d2lsbCBiZQpjbGFyaWZpZWQgd2l0aCByZXNwZWN0IHRvIFNWRSwgc28gaXQgc2FmZXN0IHRvIGFz c3VtZSB0aGF0IHRoZQpwcmVkaWNhdGUgcmVnaXN0ZXJzIGFuZCBGRlIgbXVzdCBiZSBzYXZlZCBh bmQgcmVzdG9yZWQgdG9vLgoKTm8gYXR0ZW1wdCBpcyBtYWRlIHRvIHJlc3RvcmUgdGhlIHJlc3Rv cmUgdGhlIHZlY3RvciBsZW5ndGggYWZ0ZXIKYSBjYWxsLCBmb3Igbm93LiAgSXQgaXMgZGVlbWVk IHJhdGhlciBpbnNhbmUgZm9yIEVGSSB0byBjaGFuZ2UgaXQsCmFuZCBjb250ZW1wb3JhcnkgRUZJ IGltcGxlbWVudGF0aW9ucyBjZXJ0YWlubHkgd29uJ3QuCgpTaWduZWQtb2ZmLWJ5OiBEYXZlIE1h cnRpbiA8RGF2ZS5NYXJ0aW5AYXJtLmNvbT4KQ2M6IEFyZCBCaWVzaGV1dmVsIDxhcmQuYmllc2hl dXZlbEBsaW5hcm8ub3JnPgoKLS0tCgpDaGFuZ2VzIHNpbmNlIHYxCi0tLS0tLS0tLS0tLS0tLS0K ClJlcXVlc3RlZCBieSBBcmQgQmllc2hldXZlbDoKCiogRml4IHVuYmFsYW5jZWQgaWZlbHNlIGJy YWNpbmcgdG8gY29uZm9ybSB0byB0aGUga2VybmVsIGNvZGluZyBzdHlsZS4KCiogTWFrZSBlZmlf c3ZlX3N0YXRlX3VzZWQgc3RhdGljLgoKQ2hhbmdlcyByZWxhdGVkIHRvIEFsZXggQmVubsOpZSdz IGNvbW1lbnRzOgoKKiBNaWdyYXRlIGF3YXkgZnJvbSBtYWdpYyBudW1iZXJzIGZvciBTVkVfVlFf QllURVMuCgpPdGhlcjoKCiogUmVuYW1lIHN2ZV9rZXJuZWxfbW9kZV9uZW9uX3NldHVwKCkgdG8g c3ZlX2VmaV9zZXR1cCgpLgpUaGUgRUZJIEZQU0lNRCBjb2RlIGlzIHNvbWV0aGluZyBzZW1pLWlu ZGVwZW5kZW50IGZyb20ga2VybmVsLW1vZGUgTkVPTgpub3csIHNvIHRoZSAibmVvbiIgaW4gdGhl IG5hbWVzIG5vIGxvbmdlciByZWFsbHkgbWFrZXMgc2Vuc2UuCgoqIE1ha2UgdGhlIEVGSSBGUFNJ TUQgc2V0dXAgY29kZSBkZXBlbmRlbnQgb24gQ09ORklHX0VGSSAoaXQncyBub3QKc3VwcG9zZWQg dG8gd29yayB3aXRoIENPTkZJR19FRkk9biBhbnl3YXkpLgotLS0KIGFyY2gvYXJtNjQva2VybmVs L2Zwc2ltZC5jIHwgNjAgKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKyst LS0tLQogMSBmaWxlIGNoYW5nZWQsIDU0IGluc2VydGlvbnMoKyksIDYgZGVsZXRpb25zKC0pCgpk aWZmIC0tZ2l0IGEvYXJjaC9hcm02NC9rZXJuZWwvZnBzaW1kLmMgYi9hcmNoL2FybTY0L2tlcm5l bC9mcHNpbWQuYwppbmRleCBkZDg5YWNmLi5mZmY5ZmNmIDEwMDY0NAotLS0gYS9hcmNoL2FybTY0 L2tlcm5lbC9mcHNpbWQuYworKysgYi9hcmNoL2FybTY0L2tlcm5lbC9mcHNpbWQuYwpAQCAtMTE4 LDExICsxMTgsMTMgQEAgc3RhdGljIGludCBzdmVfZGVmYXVsdF92bCA9IC0xOwogaW50IF9fcm9f YWZ0ZXJfaW5pdCBzdmVfbWF4X3ZsID0gLTE7CiAvKiBTZXQgb2YgYXZhaWxhYmxlIHZlY3RvciBs ZW5ndGhzLCBhcyB2cV90b19iaXQodnEpOiAqLwogc3RhdGljIF9fcm9fYWZ0ZXJfaW5pdCBERUNM QVJFX0JJVE1BUChzdmVfdnFfbWFwLCBTVkVfVlFfTUFYKTsKK3N0YXRpYyB2b2lkIF9fcGVyY3B1 ICplZmlfc3ZlX3N0YXRlOwogCiAjZWxzZSAvKiAhIENPTkZJR19BUk02NF9TVkUgKi8KIAogLyog RHVtbXkgZGVjbGFyYXRpb24gZm9yIGNvZGUgdGhhdCB3aWxsIGJlIG9wdGltaXNlZCBvdXQ6ICov CiBleHRlcm4gX19yb19hZnRlcl9pbml0IERFQ0xBUkVfQklUTUFQKHN2ZV92cV9tYXAsIFNWRV9W UV9NQVgpOworZXh0ZXJuIHZvaWQgX19wZXJjcHUgKmVmaV9zdmVfc3RhdGU7CiAKICNlbmRpZiAv KiAhIENPTkZJR19BUk02NF9TVkUgKi8KIApAQCAtNDQ3LDYgKzQ0OSwyMyBAQCBpbnQgc3ZlX3Zl cmlmeV92cV9tYXAodm9pZCkKIAlyZXR1cm4gcmV0OwogfQogCitzdGF0aWMgdm9pZCBfX2luaXQg c3ZlX2VmaV9zZXR1cCh2b2lkKQoreworCWlmICghSVNfRU5BQkxFRChDT05GSUdfRUZJKSkKKwkJ cmV0dXJuOworCisJLyoKKwkgKiBhbGxvY19wZXJjcHUoKSB3YXJucyBhbmQgcHJpbnRzIGEgYmFj a3RyYWNlIGlmIHRoaXMgZ29lcyB3cm9uZy4KKwkgKiBUaGlzIGlzIGV2aWRlbmNlIG9mIGEgY3Jp cHBsZWQgc3lzdGVtIGFuZCB3ZSBhcmUgcmV0dXJuaW5nIHZvaWQsCisJICogc28gbm8gYXR0ZW1w dCBpcyBtYWRlIHRvIGhhbmRsZSB0aGlzIHNpdHVhdGlvbiBoZXJlLgorCSAqLworCUJVR19PTigh c3ZlX3ZsX3ZhbGlkKHN2ZV9tYXhfdmwpKTsKKwllZmlfc3ZlX3N0YXRlID0gX19hbGxvY19wZXJj cHUoCisJCVNWRV9TSUdfUkVHU19TSVpFKHN2ZV92cV9mcm9tX3ZsKHN2ZV9tYXhfdmwpKSwgU1ZF X1ZRX0JZVEVTKTsKKwlpZiAoIWVmaV9zdmVfc3RhdGUpCisJCXBhbmljKCJDYW5ub3QgYWxsb2Nh dGUgcGVyY3B1IG1lbW9yeSBmb3IgRUZJIFNWRSBzYXZlL3Jlc3RvcmUiKTsKK30KKwogdm9pZCBf X2luaXQgc3ZlX3NldHVwKHZvaWQpCiB7CiAJdTY0IHpjcjsKQEAgLTQ4Miw2ICs1MDEsOCBAQCB2 b2lkIF9faW5pdCBzdmVfc2V0dXAodm9pZCkKIAkJc3ZlX21heF92bCk7CiAJcHJfaW5mbygiU1ZF OiBkZWZhdWx0IHZlY3RvciBsZW5ndGggJXUgYnl0ZXMgcGVyIHZlY3RvclxuIiwKIAkJc3ZlX2Rl ZmF1bHRfdmwpOworCisJc3ZlX2VmaV9zZXR1cCgpOwogfQogCiB2b2lkIGZwc2ltZF9yZWxlYXNl X3RocmVhZChzdHJ1Y3QgdGFza19zdHJ1Y3QgKmRlYWRfdGFzaykKQEAgLTc4Myw2ICs4MDQsNyBA QCBFWFBPUlRfU1lNQk9MKGtlcm5lbF9uZW9uX2VuZCk7CiAKIHN0YXRpYyBERUZJTkVfUEVSX0NQ VShzdHJ1Y3QgZnBzaW1kX3N0YXRlLCBlZmlfZnBzaW1kX3N0YXRlKTsKIHN0YXRpYyBERUZJTkVf UEVSX0NQVShib29sLCBlZmlfZnBzaW1kX3N0YXRlX3VzZWQpOworc3RhdGljIERFRklORV9QRVJf Q1BVKGJvb2wsIGVmaV9zdmVfc3RhdGVfdXNlZCk7CiAKIC8qCiAgKiBFRkkgcnVudGltZSBzZXJ2 aWNlcyBzdXBwb3J0IGZ1bmN0aW9ucwpAQCAtODA4LDEwICs4MzAsMjQgQEAgdm9pZCBfX2VmaV9m cHNpbWRfYmVnaW4odm9pZCkKIAogCVdBUk5fT04ocHJlZW1wdGlibGUoKSk7CiAKLQlpZiAobWF5 X3VzZV9zaW1kKCkpCisJaWYgKG1heV91c2Vfc2ltZCgpKSB7CiAJCWtlcm5lbF9uZW9uX2JlZ2lu KCk7Ci0JZWxzZSB7Ci0JCWZwc2ltZF9zYXZlX3N0YXRlKHRoaXNfY3B1X3B0cigmZWZpX2Zwc2lt ZF9zdGF0ZSkpOworCX0gZWxzZSB7CisJCS8qCisJCSAqIElmICFlZmlfc3ZlX3N0YXRlLCBTVkUg Y2FuJ3QgYmUgaW4gdXNlIHlldCBhbmQgZG9lc24ndCBuZWVkCisJCSAqIHByZXNlcnZpbmc6CisJ CSAqLworCQlpZiAoc3lzdGVtX3N1cHBvcnRzX3N2ZSgpICYmIGxpa2VseShlZmlfc3ZlX3N0YXRl KSkgeworCQkJY2hhciAqc3ZlX3N0YXRlID0gdGhpc19jcHVfcHRyKGVmaV9zdmVfc3RhdGUpOwor CisJCQlfX3RoaXNfY3B1X3dyaXRlKGVmaV9zdmVfc3RhdGVfdXNlZCwgdHJ1ZSk7CisKKwkJCXN2 ZV9zYXZlX3N0YXRlKHN2ZV9zdGF0ZSArIHN2ZV9mZnJfb2Zmc2V0KHN2ZV9tYXhfdmwpLAorCQkJ CSAgICAgICAmdGhpc19jcHVfcHRyKCZlZmlfZnBzaW1kX3N0YXRlKS0+ZnBzcik7CisJCX0gZWxz ZSB7CisJCQlmcHNpbWRfc2F2ZV9zdGF0ZSh0aGlzX2NwdV9wdHIoJmVmaV9mcHNpbWRfc3RhdGUp KTsKKwkJfQorCiAJCV9fdGhpc19jcHVfd3JpdGUoZWZpX2Zwc2ltZF9zdGF0ZV91c2VkLCB0cnVl KTsKIAl9CiB9CkBAIC04MjQsMTAgKzg2MCwyMiBAQCB2b2lkIF9fZWZpX2Zwc2ltZF9lbmQodm9p ZCkKIAlpZiAoIXN5c3RlbV9zdXBwb3J0c19mcHNpbWQoKSkKIAkJcmV0dXJuOwogCi0JaWYgKF9f dGhpc19jcHVfeGNoZyhlZmlfZnBzaW1kX3N0YXRlX3VzZWQsIGZhbHNlKSkKLQkJZnBzaW1kX2xv YWRfc3RhdGUodGhpc19jcHVfcHRyKCZlZmlfZnBzaW1kX3N0YXRlKSk7Ci0JZWxzZQorCWlmICgh X190aGlzX2NwdV94Y2hnKGVmaV9mcHNpbWRfc3RhdGVfdXNlZCwgZmFsc2UpKSB7CiAJCWtlcm5l bF9uZW9uX2VuZCgpOworCX0gZWxzZSB7CisJCWlmIChzeXN0ZW1fc3VwcG9ydHNfc3ZlKCkgJiYK KwkJICAgIGxpa2VseShfX3RoaXNfY3B1X3JlYWQoZWZpX3N2ZV9zdGF0ZV91c2VkKSkpIHsKKwkJ CWNoYXIgY29uc3QgKnN2ZV9zdGF0ZSA9IHRoaXNfY3B1X3B0cihlZmlfc3ZlX3N0YXRlKTsKKwor CQkJc3ZlX2xvYWRfc3RhdGUoc3ZlX3N0YXRlICsgc3ZlX2Zmcl9vZmZzZXQoc3ZlX21heF92bCks CisJCQkJICAgICAgICZ0aGlzX2NwdV9wdHIoJmVmaV9mcHNpbWRfc3RhdGUpLT5mcHNyLAorCQkJ CSAgICAgICBzdmVfdnFfZnJvbV92bChzdmVfZ2V0X3ZsKCkpIC0gMSk7CisKKwkJCV9fdGhpc19j cHVfd3JpdGUoZWZpX3N2ZV9zdGF0ZV91c2VkLCBmYWxzZSk7CisJCX0gZWxzZSB7CisJCQlmcHNp bWRfbG9hZF9zdGF0ZSh0aGlzX2NwdV9wdHIoJmVmaV9mcHNpbWRfc3RhdGUpKTsKKwkJfQorCX0K IH0KIAogI2VuZGlmIC8qIENPTkZJR19LRVJORUxfTU9ERV9ORU9OICovCi0tIAoyLjEuNAoKX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18Ka3ZtYXJtIG1haWxp bmcgbGlzdAprdm1hcm1AbGlzdHMuY3MuY29sdW1iaWEuZWR1Cmh0dHBzOi8vbGlzdHMuY3MuY29s dW1iaWEuZWR1L21haWxtYW4vbGlzdGluZm8va3ZtYXJtCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from foss.arm.com ([217.140.101.70]:58670 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752007AbdHaRBv (ORCPT ); Thu, 31 Aug 2017 13:01:51 -0400 From: Dave Martin Subject: [PATCH v2 18/28] arm64/sve: Preserve SVE registers around EFI runtime service calls Date: Thu, 31 Aug 2017 18:00:50 +0100 Message-ID: <1504198860-12951-19-git-send-email-Dave.Martin@arm.com> In-Reply-To: <1504198860-12951-1-git-send-email-Dave.Martin@arm.com> References: <1504198860-12951-1-git-send-email-Dave.Martin@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-arch-owner@vger.kernel.org List-ID: To: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas , Will Deacon , Ard Biesheuvel , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Szabolcs Nagy , Richard Sandiford , kvmarm@lists.cs.columbia.edu, libc-alpha@sourceware.org, linux-arch@vger.kernel.org Message-ID: <20170831170050.1VcY7CrJ72XaQ33UbY0fqIVJym3HDM_sMZKoANV9cCc@z> The EFI runtime services ABI allows EFI to make free use of the FPSIMD registers during EFI runtime service calls, subject to the callee-save requirements of the AArch64 procedure call standard. However, the SVE architecture allows upper bits of the SVE vector registers to be zeroed as a side-effect of FPSIMD V-register writes. This means that the SVE vector registers must be saved in their entirety in order to avoid data loss: non-SVE-aware EFI implementations cannot restore them correctly. The non-IRQ case is already handled gracefully by kernel_neon_begin(). For the IRQ case, this patch allocates a suitable per-CPU stash buffer for the full SVE register state and uses it to preserve the affected registers around EFI calls. It is currently unclear how the EFI runtime services ABI will be clarified with respect to SVE, so it safest to assume that the predicate registers and FFR must be saved and restored too. No attempt is made to restore the restore the vector length after a call, for now. It is deemed rather insane for EFI to change it, and contemporary EFI implementations certainly won't. Signed-off-by: Dave Martin Cc: Ard Biesheuvel --- Changes since v1 ---------------- Requested by Ard Biesheuvel: * Fix unbalanced ifelse bracing to conform to the kernel coding style. * Make efi_sve_state_used static. Changes related to Alex Bennée's comments: * Migrate away from magic numbers for SVE_VQ_BYTES. Other: * Rename sve_kernel_mode_neon_setup() to sve_efi_setup(). The EFI FPSIMD code is something semi-independent from kernel-mode NEON now, so the "neon" in the names no longer really makes sense. * Make the EFI FPSIMD setup code dependent on CONFIG_EFI (it's not supposed to work with CONFIG_EFI=n anyway). --- arch/arm64/kernel/fpsimd.c | 60 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 54 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index dd89acf..fff9fcf 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -118,11 +118,13 @@ static int sve_default_vl = -1; int __ro_after_init sve_max_vl = -1; /* Set of available vector lengths, as vq_to_bit(vq): */ static __ro_after_init DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); +static void __percpu *efi_sve_state; #else /* ! CONFIG_ARM64_SVE */ /* Dummy declaration for code that will be optimised out: */ extern __ro_after_init DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); +extern void __percpu *efi_sve_state; #endif /* ! CONFIG_ARM64_SVE */ @@ -447,6 +449,23 @@ int sve_verify_vq_map(void) return ret; } +static void __init sve_efi_setup(void) +{ + if (!IS_ENABLED(CONFIG_EFI)) + return; + + /* + * alloc_percpu() warns and prints a backtrace if this goes wrong. + * This is evidence of a crippled system and we are returning void, + * so no attempt is made to handle this situation here. + */ + BUG_ON(!sve_vl_valid(sve_max_vl)); + efi_sve_state = __alloc_percpu( + SVE_SIG_REGS_SIZE(sve_vq_from_vl(sve_max_vl)), SVE_VQ_BYTES); + if (!efi_sve_state) + panic("Cannot allocate percpu memory for EFI SVE save/restore"); +} + void __init sve_setup(void) { u64 zcr; @@ -482,6 +501,8 @@ void __init sve_setup(void) sve_max_vl); pr_info("SVE: default vector length %u bytes per vector\n", sve_default_vl); + + sve_efi_setup(); } void fpsimd_release_thread(struct task_struct *dead_task) @@ -783,6 +804,7 @@ EXPORT_SYMBOL(kernel_neon_end); static DEFINE_PER_CPU(struct fpsimd_state, efi_fpsimd_state); static DEFINE_PER_CPU(bool, efi_fpsimd_state_used); +static DEFINE_PER_CPU(bool, efi_sve_state_used); /* * EFI runtime services support functions @@ -808,10 +830,24 @@ void __efi_fpsimd_begin(void) WARN_ON(preemptible()); - if (may_use_simd()) + if (may_use_simd()) { kernel_neon_begin(); - else { - fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state)); + } else { + /* + * If !efi_sve_state, SVE can't be in use yet and doesn't need + * preserving: + */ + if (system_supports_sve() && likely(efi_sve_state)) { + char *sve_state = this_cpu_ptr(efi_sve_state); + + __this_cpu_write(efi_sve_state_used, true); + + sve_save_state(sve_state + sve_ffr_offset(sve_max_vl), + &this_cpu_ptr(&efi_fpsimd_state)->fpsr); + } else { + fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state)); + } + __this_cpu_write(efi_fpsimd_state_used, true); } } @@ -824,10 +860,22 @@ void __efi_fpsimd_end(void) if (!system_supports_fpsimd()) return; - if (__this_cpu_xchg(efi_fpsimd_state_used, false)) - fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state)); - else + if (!__this_cpu_xchg(efi_fpsimd_state_used, false)) { kernel_neon_end(); + } else { + if (system_supports_sve() && + likely(__this_cpu_read(efi_sve_state_used))) { + char const *sve_state = this_cpu_ptr(efi_sve_state); + + sve_load_state(sve_state + sve_ffr_offset(sve_max_vl), + &this_cpu_ptr(&efi_fpsimd_state)->fpsr, + sve_vq_from_vl(sve_get_vl()) - 1); + + __this_cpu_write(efi_sve_state_used, false); + } else { + fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state)); + } + } } #endif /* CONFIG_KERNEL_MODE_NEON */ -- 2.1.4