From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4EDCCC433FE for ; Sat, 5 Nov 2022 00:02:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xuBKNCk8dEQPDwWzpB/geD1bWRVW9vg0ong+TZYzPAY=; b=Y2ptIuBdZ92aiY HWX0Amol0QqtbgjF+xeHVa/bMbsXVbfGfvxfe0REeLpN+s5GbbbEt2XiLG+FaoExkA9ziwRiXf3W0 fUJEeM8wj1OnFdV3AkrtvmHF+BBITbdlPAlRBJzy/RwPiY/XX9ZsQFyL4oi5vyF4IaMt2qOjlh4QC b+fTaODGi8zttD6kWHnBQA08biW8pVaXBPfGIciQ1ofmgNrt4qISjkmnkQi7PNPvJ731BJynEwaWT rr30s+PCyovDTYT9Mz85yqLLDlN23VfeuFVj6dlik6mox3qSHfJHStuN2XodWYsCGWaXicclDviZi gxIX6hlqzy+4lgHmD67w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1or6d3-005Zck-Ia; Sat, 05 Nov 2022 00:01:41 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1or6XG-005Wno-5x for linux-arm-kernel@lists.infradead.org; Fri, 04 Nov 2022 23:55:44 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id BEF95B82FF0; Fri, 4 Nov 2022 23:55:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5122AC43470; Fri, 4 Nov 2022 23:55:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1667606139; bh=gsTnt56/AFl8UwpDndzR9Ny2/yGa983cE7e0tXBK27o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SclOCnO5XzBJE5TgkTilb2WjYiLY8vMt+P5Q8PUyatonjXmT5y9Z5JODWkghQWSeq xBd3b2f9OMXwYUr4auC80d0smJYiV3lohwyOzXOc4Gaeu0dlDNwRaXpWsLA2Vk+OF/ JE6AXzpOwXt4fwSdoxcI8TxSdII/C3CaF1uID3sDqL7k5EGJncf2LMoYfYsuilZL2D mWE73Ozzq2cJnozegwyEy9RombitdaoHwfrCYw6Vnt7Q/j2r9M1dXFbuO0gaALZXon dWrtKT88tmOq9Pj1XwYMnHlRMe84F88G1naMmzHFDyh3doc0jKSK6D6wM6Jwx+HBGq O5LxQjDdiJlrQ== From: Mark Brown To: Catalin Marinas , Will Deacon , Marc Zyngier Cc: Lorenzo Pieralisi , Mark Rutland , Sami Mujawar , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Mark Brown Subject: [PATCH v1 16/18] arm64/nmi: Add handling of superpriority interrupts as NMIs Date: Fri, 4 Nov 2022 23:54:51 +0000 Message-Id: <20221104235453.870573-17-broonie@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221104235453.870573-1-broonie@kernel.org> References: <20221104235453.870573-1-broonie@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7602; i=broonie@kernel.org; h=from:subject; bh=gsTnt56/AFl8UwpDndzR9Ny2/yGa983cE7e0tXBK27o=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBjZaZLNkmGU986rCYX0EcdQxpNY8X7YE2Ohr5kbP9F DyUZEJOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCY2WmSwAKCRAk1otyXVSH0PEaB/ 4tyeag0xV5FfUulmua59NDIeBfRO5rli4EjcX5Hqibw774vaF3fQoiEz/Fy1gubiQrBUlQOiFn0J+W WEAwgHOcBFPGxRH9HP23BNVF95EYAdQ72bNU4b6a3/3GkXKit968Xio7i6Qu89IPy+KWo617uHsR3V MnA3HUAKq/PLPPf5IUwrcGeUgchgmHu8uh/lUorBO6lnKPTnOGbCKRyRQTWHYaq7W4beB6T1Q/Ykqe zHfYvqhSqIXtTR7/O6W71CE62dt47O7GfHNMJrAm02P7Hdb/UEVxaCk9lVj9sGcNKrA6ti9Sg0ZMjY 2o2zwTbU5uckjMCksEvbncHFh0L8q+ X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221104_165542_553182_BE177422 X-CRM114-Status: GOOD ( 18.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Our goal with superpriority interrupts is to use them as NMIs, taking advantage of the much smaller regions where they are masked to allow prompt handling of the most time critical interrupts. When an interrupt configured with superpriority we will enter EL1 as normal for any interrupt, the presence of a superpriority interrupt is indicated with a status bit in ISR_EL1. We use this to check for the presence of a superpriority interrupt before we unmask anything in elX_interrupt(), reporting without unmasking any interrupts. If no superpriority interrupt is present then we unmask superpriority interrupts and proceed with normal interrupt handling. Both IRQs and FIQs may be configured with superpriority so we handle both, passing an additional root handler into the elX_interrupt() function along with the mask for the bit in ISR_EL1 which indicates the relevant kind of superpriority interrupt. These root handlers can be configured by the interrupt controller similarly to the root handlers for normal interrupts using the newly added set_handle_nmi_irq() and set_handle_nmi_fiq() functions. Signed-off-by: Mark Brown --- arch/arm64/include/asm/irq.h | 2 + arch/arm64/kernel/entry-common.c | 64 +++++++++++++++++++++++++++----- arch/arm64/kernel/irq.c | 32 ++++++++++++++++ 3 files changed, 88 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h index fac08e18bcd5..2ab05d899bf6 100644 --- a/arch/arm64/include/asm/irq.h +++ b/arch/arm64/include/asm/irq.h @@ -8,6 +8,8 @@ struct pt_regs; +int set_handle_nmi_irq(void (*handle_irq)(struct pt_regs *)); +int set_handle_nmi_fiq(void (*handle_fiq)(struct pt_regs *)); int set_handle_irq(void (*handle_irq)(struct pt_regs *)); #define set_handle_irq set_handle_irq int set_handle_fiq(void (*handle_fiq)(struct pt_regs *)); diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 32547723fcc8..006ccf2f1def 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -279,6 +279,8 @@ static void do_interrupt_handler(struct pt_regs *regs, set_irq_regs(old_regs); } +extern void (*handle_arch_nmi_irq)(struct pt_regs *); +extern void (*handle_arch_nmi_fiq)(struct pt_regs *); extern void (*handle_arch_irq)(struct pt_regs *); extern void (*handle_arch_fiq)(struct pt_regs *); @@ -458,6 +460,15 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs) } } +static __always_inline void __el1_nmi(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ + arm64_enter_nmi(regs); + do_interrupt_handler(regs, handler); + arm64_exit_nmi(regs); + _allint_clear(); +} + static __always_inline void __el1_pnmi(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { @@ -479,10 +490,21 @@ static __always_inline void __el1_irq(struct pt_regs *regs, exit_to_kernel_mode(regs); } -static void noinstr el1_interrupt(struct pt_regs *regs, - void (*handler)(struct pt_regs *)) + +static void noinstr el1_interrupt(struct pt_regs *regs, u64 nmi_flag, + void (*handler)(struct pt_regs *), + void (*nmi_handler)(struct pt_regs *)) { - nmi_unmask(); + if (system_uses_nmi()) { + /* Is there a NMI to handle? */ + if (read_sysreg(isr_el1) & nmi_flag) { + __el1_nmi(regs, nmi_handler); + return; + } + + _allint_clear(); + } + write_sysreg(DAIF_PROCCTX_NOIRQ, daif); if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) @@ -493,12 +515,12 @@ static void noinstr el1_interrupt(struct pt_regs *regs, asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs) { - el1_interrupt(regs, handle_arch_irq); + el1_interrupt(regs, ISR_EL1_IS, handle_arch_irq, handle_arch_nmi_irq); } asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs) { - el1_interrupt(regs, handle_arch_fiq); + el1_interrupt(regs, ISR_EL1_FS, handle_arch_fiq, handle_arch_nmi_fiq); } asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs) @@ -710,12 +732,34 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs) } } -static void noinstr el0_interrupt(struct pt_regs *regs, - void (*handler)(struct pt_regs *)) +static void noinstr el0_interrupt(struct pt_regs *regs, u64 nmi_flag, + void (*handler)(struct pt_regs *), + void (*nmi_handler)(struct pt_regs *)) { enter_from_user_mode(regs); - nmi_unmask(); + if (system_uses_nmi()) { + /* Is there a NMI to handle? */ + if (read_sysreg(isr_el1) & nmi_flag) { + /* + * Any system with FEAT_NMI should not be + * affected by Spectre v2 so we don't mitigate + * here. + */ + + arm64_enter_nmi(regs); + do_interrupt_handler(regs, nmi_handler); + arm64_exit_nmi(regs); + + _allint_clear(); + + exit_to_user_mode(regs); + return; + } + + _allint_clear(); + } + write_sysreg(DAIF_PROCCTX_NOIRQ, daif); if (regs->pc & BIT(55)) @@ -730,7 +774,7 @@ static void noinstr el0_interrupt(struct pt_regs *regs, static void noinstr __el0_irq_handler_common(struct pt_regs *regs) { - el0_interrupt(regs, handle_arch_irq); + el0_interrupt(regs, ISR_EL1_IS, handle_arch_irq, handle_arch_nmi_irq); } asmlinkage void noinstr el0t_64_irq_handler(struct pt_regs *regs) @@ -740,7 +784,7 @@ asmlinkage void noinstr el0t_64_irq_handler(struct pt_regs *regs) static void noinstr __el0_fiq_handler_common(struct pt_regs *regs) { - el0_interrupt(regs, handle_arch_fiq); + el0_interrupt(regs, ISR_EL1_FS, handle_arch_fiq, handle_arch_nmi_fiq); } asmlinkage void noinstr el0t_64_fiq_handler(struct pt_regs *regs) diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index 38dbd3828f13..77a1ea90b244 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -85,6 +85,16 @@ void do_softirq_own_stack(void) } #endif +static void default_handle_nmi_irq(struct pt_regs *regs) +{ + panic("Superpriority IRQ taken without a root NMI IRQ handler\n"); +} + +static void default_handle_nmi_fiq(struct pt_regs *regs) +{ + panic("Superpriority FIQ taken without a root NMI FIQ handler\n"); +} + static void default_handle_irq(struct pt_regs *regs) { panic("IRQ taken without a root IRQ handler\n"); @@ -95,9 +105,31 @@ static void default_handle_fiq(struct pt_regs *regs) panic("FIQ taken without a root FIQ handler\n"); } +void (*handle_arch_nmi_irq)(struct pt_regs *) __ro_after_init = default_handle_nmi_irq; +void (*handle_arch_nmi_fiq)(struct pt_regs *) __ro_after_init = default_handle_nmi_fiq; void (*handle_arch_irq)(struct pt_regs *) __ro_after_init = default_handle_irq; void (*handle_arch_fiq)(struct pt_regs *) __ro_after_init = default_handle_fiq; +int __init set_handle_nmi_irq(void (*handle_nmi_irq)(struct pt_regs *)) +{ + if (handle_arch_nmi_irq != default_handle_nmi_irq) + return -EBUSY; + + handle_arch_nmi_irq = handle_nmi_irq; + pr_info("Root superpriority IRQ handler: %ps\n", handle_nmi_irq); + return 0; +} + +int __init set_handle_nmi_fiq(void (*handle_nmi_fiq)(struct pt_regs *)) +{ + if (handle_arch_nmi_fiq != default_handle_nmi_fiq) + return -EBUSY; + + handle_arch_nmi_fiq = handle_nmi_fiq; + pr_info("Root superpriority FIQ handler: %ps\n", handle_nmi_fiq); + return 0; +} + int __init set_handle_irq(void (*handle_irq)(struct pt_regs *)) { if (handle_arch_irq != default_handle_irq) -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel