From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6B533C61D97 for ; Thu, 23 Nov 2023 17:45:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=VxmfqvlUgMZ3KJf5qf28MFPyqS0cTiT9g44MTZfF3zI=; b=Z7ocde5/1e/rNgVTK0VTmJtz4U 72Hk0/lhOa+QQHk+pTTtW4NjGQCMxf/gAqBe7DWmYb2dArTEso7TzlVwO0aQ8AIdI6Tyb14wEqZUe 22ydOg70srwqglq8yIdngiudJTE8CdsNB9zRTTbX80fE+/w1jr2yndt7xRG27bOZUmTEywQbQ3KPO HjP8OQct4Vk4GAGe0neyoY1SeQXlTmaEkIUm/J/Iw3CLsPhvacCJph6pEh0UuK0ZrYyXnrYt7YnkF ccZTKrZD9c/WRCJI4auFyyHD1vS0sBB3mbVgGP8IRvME0/pq+7COzJbUInNCioFW6b81HyPsQwrGG CdGVBYzg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6DlC-005SJm-06; Thu, 23 Nov 2023 17:45:06 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6Dl1-005SDh-0B for linux-arm-kernel@lists.infradead.org; Thu, 23 Nov 2023 17:44:56 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5cd1172b815so7290287b3.0 for ; Thu, 23 Nov 2023 09:44:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700761490; x=1701366290; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IORFzrnhmH8pTtJZq6QUcnNLQ4IOA6WwMeoD6YxYFMU=; b=ubS5t89JKajRiy25y0Ggzzh5WypPg+xKN2pIuCxyKb95F/bGgVPPYFjk9CnQnHDeLF 4HIXHjUo4rx3saMdHmAMgs9/aLKhShdluMo7Z3tfprGjeINQIWQkTh/NZp/BeM54AdaJ 3BzfxbtwgyQacrKqdG/+VoOFOFmjHr4QWvaQV3LXTQ/w9zWFFj56hPlcb9E3XsPmxtL8 EgXtuGdH5s7VchqFitfAeXwbFTz0ziQyTmfwQ+jYD8tC1t9Bkn61u7AccQRmnA4bj59t 8r+VlVuH7tZa8X1CbRvFyoWfmkpq/BwM5KKJL3+QRjMMXutfODyQView9CqxakNZ/GUA FqIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700761490; x=1701366290; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IORFzrnhmH8pTtJZq6QUcnNLQ4IOA6WwMeoD6YxYFMU=; b=Tc67Oelr5ejIaUQEv40u0ZpXTfqaYisZmtJaYsW5+LS4IAbI62P5goAB1oTLpxpKZS 3ygXcy26XxihYpHXoGe9n9UV6nfFYavhJ+lqLzhkzPS5sh/4wyzVxHcoUDFaO7xavhxg MXDi5MqIJkgVN3+eIUwx8koZlHBFuTjxrVHR1arzwqd39Z18zFFXWjIchhl49GhGk6ZD MfrVRwBHPBmIaUkQmhdau2agdaRHBJv52fqp+1JpiHGHBWg50nsMmh6Tu5gGv1YwpawP ybpHZ/8bYaPdC8c1FKSPJFjIsYLF42Wcd5YNtJhHJ1hufbMiJtScgSEJxt/Z5cjLT9wg b76w== X-Gm-Message-State: AOJu0Yz0FZr5IQ4JuRmvp+SmOqYWjqB8VjBuIRAtb/bipJ/FrmsbMQzn 4AfkzHO/vk53+aiQynyN+ENihsbwrcSwKHOLjq3Se8+Jee2MS1nGsGqb3RdqSRfoHAqJXMLl+37 r5gGxBkVrLhx6V39GZ4q+yoTfV6pMozxVjntCXFnDXjaLvL9qBwuUOJArR7oMooSaOuhopVpAII w= X-Google-Smtp-Source: AGHT+IENyA+lCF0qwoJNsYqhXB3rxhQfWADU2UKJnxbfnjtPd/JKao59vS7CDU36uBXyq13Y/i2vCqMm X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:2f08:b0:5be:baac:54e4 with SMTP id ev8-20020a05690c2f0800b005bebaac54e4mr97448ywb.5.1700761490424; Thu, 23 Nov 2023 09:44:50 -0800 (PST) Date: Thu, 23 Nov 2023 18:44:37 +0100 In-Reply-To: <20231123174433.737171-6-ardb@google.com> Mime-Version: 1.0 References: <20231123174433.737171-6-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2715; i=ardb@kernel.org; h=from:subject; bh=XgUJvgH7rJxp04IZGUacpXjiLeZAe4Tm3e6D/Uz4W44=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITW+v1XE6u6hKceWHBJ3ty/+9bb2ifmqThuXquRtWw5lF CuHVXJ1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIm0CzMyHC2cnnxBt2de2O/f 2o/Tfv/Oe8gbcNbmyJ4Ja+e0c9363s/wP8TE+LjMjU3VG3uTFn5nU5i5W/+8tarHVPOfM2dFnlR 5ygYA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231123174433.737171-9-ardb@google.com> Subject: [PATCH v2 3/4] arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231123_094455_095464_66E11B01 X-CRM114-Status: GOOD ( 17.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that kernel mode FPSIMD state is context switched along with other task state, we can enable the existing logic that keeps track of which task's FPSIMD state the CPU is holding in its registers. If it is the context of the task that we are switching to, we can elide the reload of the FPSIMD state from memory. Note that we also need to check whether the FPSIMD state on this CPU is the most recent: if a task gets migrated away and back again, the state in memory may be more recent than the state in the CPU. So add another CPU id field to task_struct to keep track of this. (We could reuse the existing CPU id field used for user mode context, but that might result in user state to be discarded unnecessarily, given that two distinct CPUs could be holding the most recent user mode state and the most recent kernel mode state) Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/processor.h | 1 + arch/arm64/kernel/fpsimd.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+) diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index dcb51c0571af..332f15d0abcf 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -169,6 +169,7 @@ struct thread_struct { struct debug_info debug; /* debugging */ struct user_fpsimd_state kmode_fpsimd_state; + unsigned int kmode_fpsimd_cpu; #ifdef CONFIG_ARM64_PTR_AUTH struct ptrauth_keys_user keys_user; #ifdef CONFIG_ARM64_PTR_AUTH_KERNEL diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index d115f9474982..e848b72dc0c8 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1476,12 +1476,30 @@ void do_fpsimd_exc(unsigned long esr, struct pt_regs *regs) static void fpsimd_load_kernel_state(struct task_struct *task) { + struct cpu_fp_state *last = this_cpu_ptr(&fpsimd_last_state); + + /* + * Elide the load if this CPU holds the most recent kernel mode + * FPSIMD context of the current task. + */ + if (last->st == &task->thread.kmode_fpsimd_state && + task->thread.kmode_fpsimd_cpu == smp_processor_id()) + return; + fpsimd_load_state(&task->thread.kmode_fpsimd_state); } static void fpsimd_save_kernel_state(struct task_struct *task) { + struct cpu_fp_state cpu_fp_state = { + .st = &task->thread.kmode_fpsimd_state, + .to_save = FP_STATE_FPSIMD, + }; + fpsimd_save_state(&task->thread.kmode_fpsimd_state); + fpsimd_bind_state_to_cpu(&cpu_fp_state); + + task->thread.kmode_fpsimd_cpu = smp_processor_id(); } void fpsimd_thread_switch(struct task_struct *next) -- 2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel