From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7DEA1C4167B for ; Mon, 27 Nov 2023 12:23:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=pAr3aftacAtXLvrtksFtNR6zCQChfoykpaDEHnxmRIk=; b=IUsWGvTcmjbeDEZJwDadAqe1W/ 1NleYppWaQhOx6TqxUSoaWJ+DuEwrHPL0wGfUkbKz28ZjV8b5EtCO7RoootDekEu2ofe8Tmy05Wgz +9lepfalWkdxrkUF1UbDqHMPv4Lnxkwsct8J60nsaP2PCdaq7tRQzc5J2XYjpUX629PwwbZaD/h+o fT16faq5lRLUYORag6nRJr3lBEdrv/c66saUOV4xUTZBN0r3wRWbOrHre726mObtpagEjEGNZRSPX 6jZdS3/wdMuKnAlZqokj0Up6Ikqsncifkr1J3xwSp+v4GhyYmwNTnLMiPpuKA6dlM1Eqjy2Yh7sgR 6xnqGi0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r7ae7-002RXw-2c; Mon, 27 Nov 2023 12:23:27 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r7ae3-002RVG-0H for linux-arm-kernel@lists.infradead.org; Mon, 27 Nov 2023 12:23:24 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-332eecd96feso1824191f8f.3 for ; Mon, 27 Nov 2023 04:23:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701087800; x=1701692600; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7RC7fcm+iIbvYs4DSiycpu+OLkiF/mkbOdoMlvtSw6k=; b=2gEVmezjGT1vAkXSW+zvggu42qd3brOT7+l3lQsVz759+Lj7LbpMs8DMKnD8lcCr8P C3uJwFTBPRC2qnwcz9BuY3y2tEa5XVqjyHMH+eq2lRw1wb8CjoKXT/exzYGeM3tE++0r Lxe4GVRbt0ucmdeO/mTWrkuRVL+cM64VqEmVHd5BoEj7mAB576G5LWCaiW4p3lywx8yQ +jzNsMOboWalxS2KAR7V4Y4gMLK0raYsASMU0QItw7Bdvl3nojMgPSk8z6WWamlvSq+i 9w0Ta+AWzh2auGj18BlrP8dEK8CbdIIG0i1wpnCPmHOOsz5kNio0wTMopvuyuD4aNnzV 6p5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701087800; x=1701692600; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7RC7fcm+iIbvYs4DSiycpu+OLkiF/mkbOdoMlvtSw6k=; b=VDEavw04GaQTuoMVZaxCq4n6NRhoAiQet49Q0k2PHt65W0gfjLkumWVUlG557k9xr7 EZ6Xyz/NukjdhwXpsx0Pph2JXGDewW5VYb/FosBfmuQs/c/91XtuuoivgaqCct/v+wEi UT8kH2x14fkIDLcnyxSpGAZ5OuqLmAWl5WxUOlpXyzTmnZUnbcFIJkcGAMocd1dkdGIn 9CTATqun0tcUZHZxfwNGjjMe9uEkgEIln3bwd2g3br1Qg9/LCi3wCC+uFl4itfVVEj62 HeXaT7WpkH9pDbTBWl3J/i00v4PzP/PR4THzocXjvuBqc7HkyVdEY9DtLEwEHGnO79R8 43+Q== X-Gm-Message-State: AOJu0YzpE/1jVJ2IeWtM11j+fdUQxuCK5aRVmE3yWxNwGuHDSgRaPlK8 Yb8Qdww73cQYeuIBCKnOVZjXDPjI8jZARmv3TX43F3t+yJca/CJ5PmoWo5BKD/q7tWoWOmVwHd1 Im5HIzNERy6vg5+Qi5WoX9kezqDi1Y4khchkJCjZDhEF7dbZr42UT3tXrlBXlBOqPWr/xuDlYCO E= X-Google-Smtp-Source: AGHT+IHf0gTC1PXtSo2Om+FUZLbT7f8NZUczw3CzhB5wimmyrlTb6QDofzUzL3kLMA0a2OUJC9vKEbW3 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:fc52:0:b0:332:cc18:b798 with SMTP id e18-20020adffc52000000b00332cc18b798mr178167wrs.14.1701087799840; Mon, 27 Nov 2023 04:23:19 -0800 (PST) Date: Mon, 27 Nov 2023 13:23:03 +0100 In-Reply-To: <20231127122259.2265164-7-ardb@google.com> Mime-Version: 1.0 References: <20231127122259.2265164-7-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2761; i=ardb@kernel.org; h=from:subject; bh=lq25ekjnEUttIKcfoZQhq0creUwzzdf4IQiZXvjxNR0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWlS73BdDabothCI+GzBy6dev+y8ujX1SE8exbv+b323 nG7x8afOkpZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEGCQZ/hklH69KkPmpnFkt t/yB3oLV3xasDbzwI4e9eXP2m+SLl74z/OGL9j/6qqbysPDhBTNOMV9XOdBd63t7IRNP9PRpoo8 XOfICAA== X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231127122259.2265164-10-ardb@google.com> Subject: [PATCH v3 3/5] arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers , Sebastian Andrzej Siewior X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231127_042323_223956_4BEE9A4E X-CRM114-Status: GOOD ( 17.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that kernel mode FPSIMD state is context switched along with other task state, we can enable the existing logic that keeps track of which task's FPSIMD state the CPU is holding in its registers. If it is the context of the task that we are switching to, we can elide the reload of the FPSIMD state from memory. Note that we also need to check whether the FPSIMD state on this CPU is the most recent: if a task gets migrated away and back again, the state in memory may be more recent than the state in the CPU. So add another CPU id field to task_struct to keep track of this. (We could reuse the existing CPU id field used for user mode context, but that might result in user state to be discarded unnecessarily, given that two distinct CPUs could be holding the most recent user mode state and the most recent kernel mode state) Signed-off-by: Ard Biesheuvel Reviewed-by: Mark Brown --- arch/arm64/include/asm/processor.h | 1 + arch/arm64/kernel/fpsimd.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+) diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index dcb51c0571af..332f15d0abcf 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -169,6 +169,7 @@ struct thread_struct { struct debug_info debug; /* debugging */ struct user_fpsimd_state kmode_fpsimd_state; + unsigned int kmode_fpsimd_cpu; #ifdef CONFIG_ARM64_PTR_AUTH struct ptrauth_keys_user keys_user; #ifdef CONFIG_ARM64_PTR_AUTH_KERNEL diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 198918805bf6..112111a078b6 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1476,12 +1476,30 @@ void do_fpsimd_exc(unsigned long esr, struct pt_regs *regs) static void fpsimd_load_kernel_state(struct task_struct *task) { + struct cpu_fp_state *last = this_cpu_ptr(&fpsimd_last_state); + + /* + * Elide the load if this CPU holds the most recent kernel mode + * FPSIMD context of the current task. + */ + if (last->st == &task->thread.kmode_fpsimd_state && + task->thread.kmode_fpsimd_cpu == smp_processor_id()) + return; + fpsimd_load_state(&task->thread.kmode_fpsimd_state); } static void fpsimd_save_kernel_state(struct task_struct *task) { + struct cpu_fp_state cpu_fp_state = { + .st = &task->thread.kmode_fpsimd_state, + .to_save = FP_STATE_FPSIMD, + }; + fpsimd_save_state(&task->thread.kmode_fpsimd_state); + fpsimd_bind_state_to_cpu(&cpu_fp_state); + + task->thread.kmode_fpsimd_cpu = smp_processor_id(); } void fpsimd_thread_switch(struct task_struct *next) -- 2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel