From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA980C433F5 for ; Mon, 15 Nov 2021 08:51:17 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 966F463219 for ; Mon, 15 Nov 2021 08:51:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 966F463219 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=glDvnTkdL5MrcppQ40bucsC9pv/RknliTPEpo0Zh9rk=; b=GpUd0AETk82CpT P4Lh9Hr2Av+cslSon9VSu/j6IM+tf+eCG9O+3gsD0hT5cMsj98MYGtrtRcEqRGmWXReb+b1Rd317D Oxd3s51fXxnjUfA7yeiGbPZd1UJfQyTg0NfuMMFVPGjgYEHss5uXBj1ZPIPZcpwAtHPjxnOeo0tW0 LG+lGhNjdvK+3DZ+m5PRcXysuXKzGWQgEG+0Gckc93+qBn41blDoHEyFb1E+5pTG8w3NjZHN7EcST 8vmxe0rPRpqLVlRmzzuJYl4bFo/6YxiutyxfM09hBmfT3QKa+lBixZjx3x5HTr/9kbPaXFizQSZZz GpbwIB3g24V0EYI53wKQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmXgP-00Ejum-QR; Mon, 15 Nov 2021 08:49:46 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmXef-00Ej8O-1O for linux-arm-kernel@lists.infradead.org; Mon, 15 Nov 2021 08:47:58 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id C08976321A; Mon, 15 Nov 2021 08:47:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1636966076; bh=DMIxGqi2+AxtywHz2lB7kdmcuLhNihi5LeTJvY9UFic=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GDQW3s9cM4IdTJM2mF3635LtlfbHSjTxLH+S2xaaSMHaq41wq2MPU2H7Gc67spxIc 5xwKaU2N5zTmM20e0PV/on/qryqWmYfEZhsCX6QnPjT4sAom1W9uB/O2WyfalSW46S eM/bBVYkUNDH/hTvbxBheK0ivEe6mQQsT472aDEv1zd++4jprA5rJVg/ktvtcyhV3A CO7aVBigAu8i/03x2c7Se+qFyUjFL37lIYQIgRU4p7I6K1QUgsmcBxOvMZZf1e/OYB x2DatrfUP5dVDxmthaIz293Cl74kgSKPBol8WE5/Q0puShAB8OJgq9Zd7EzrI4+M7n 8UdYPKL0hbRwQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Russell King , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers , Nicolas Pitre Subject: [PATCH v4 7/9] ARM: implement IRQ stacks Date: Mon, 15 Nov 2021 09:47:30 +0100 Message-Id: <20211115084732.3704393-8-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211115084732.3704393-1-ardb@kernel.org> References: <20211115084732.3704393-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7727; h=from:subject; bh=DMIxGqi2+AxtywHz2lB7kdmcuLhNihi5LeTJvY9UFic=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhkh6gi2Mz1gRen4wWbTCVPt1/cIdd2oEqykVM9M5N xzSIA5aJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYZIeoAAKCRDDTyI5ktmPJJ7fDA CCsGIZXK2pXRD1pQatAiuIbeHXAVQdVw8JLLFCQUbY9nGiEQXT0ieyoTy/R7dFAdt9JaDVtkDved8R Fc3kYUu+sTdnGdfbYztA5esqQlF/ZOAMiewtlGnV53Nym4LtYkta7GyGx2M6YMvckDX2fz/mL0uc9d MKcm7yurP0z1UuY/kynAzBuMQE6lGsFczPVVEKvB4ZVHgHBYWPRzwsBvOq+9yN8/2J51kxenXYj4w3 4kP3apOWKQarJAjrvzkSMD/ZRtxX3volet7W3h5Fpj0fQb7lw7WvzM5YAy7CUVwdk+Mxrw8DFndbv4 IM35jmcJxH8kqpynncWtRxaa9/GNYFs+LN49fq6XFVefnXVa0S96bY1PzwEo/zjWtWlQFV/VHatsL1 Lv8iywmYhVXBC1YhY69JHP0IdaVoGsL262MHL4b8c5+7U6jc2mJKDbi1kjjbU0kU2X5RONHFdlen2F 54QwE2ChfCrLuCj/q20vDGkwr2pKS0XU/ERMkqOwcGsSM= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211115_004757_158778_2DCB3F62 X-CRM114-Status: GOOD ( 24.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well. Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet. Signed-off-by: Ard Biesheuvel Acked-by: Linus Walleij Tested-by: Keith Packard Acked-by: Nick Desaulniers --- arch/arm/Kconfig | 4 ++ arch/arm/include/asm/assembler.h | 4 ++ arch/arm/kernel/entry-armv.S | 54 ++++++++++++++++++-- arch/arm/kernel/irq.c | 23 +++++++++ arch/arm/kernel/traps.c | 15 +++++- arch/arm/lib/backtrace-clang.S | 8 +++ arch/arm/lib/backtrace.S | 8 +++ 7 files changed, 112 insertions(+), 4 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index f0f9e8bec83a..aa421f29910b 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1166,6 +1166,10 @@ config CURRENT_POINTER_IN_TPIDRURO def_bool y depends on SMP && CPU_32v6K && !CPU_V6 +config IRQSTACKS + def_bool y + depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK + config ARM_CPU_TOPOLOGY bool "Support cpu topology definition" depends on SMP && CPU_V7 diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 870bfaea4318..1b9d4df331aa 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -86,6 +86,10 @@ #define IMM12_MASK 0xfff +/* the frame pointer used for stack unwinding */ +ARM( fpreg .req r11 ) +THUMB( fpreg .req r7 ) + /* * Enable and disable interrupts */ diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index deff286eb5ea..1c7590eef712 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -36,10 +36,58 @@ /* * Interrupt handling. */ - .macro irq_handler + .macro irq_handler, from_user:req #ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER mov r0, sp +#ifdef CONFIG_IRQSTACKS + mov_l r2, irq_stack_ptr @ Take base address + mrc p15, 0, r3, c13, c0, 4 @ Get CPU offset +#ifdef CONFIG_UNWINDER_ARM + mov fpreg, sp @ Preserve original SP +#else + mov r8, fp @ Preserve original FP + mov r9, sp @ Preserve original SP +#endif + ldr sp, [r2, r3] @ Load SP from per-CPU var + .if \from_user == 0 +UNWIND( .setfp fpreg, sp ) + @ + @ If we took the interrupt while running in the kernel, we may already + @ be using the IRQ stack, so revert to the original value in that case. + @ + subs r2, sp, r0 @ SP above bottom of IRQ stack? + rsbscs r2, r2, #THREAD_SIZE @ ... and below the top? + movcs sp, r0 @ If so, revert to incoming SP + +#ifndef CONFIG_UNWINDER_ARM + @ + @ Inform the frame pointer unwinder where the next frame lives + @ + movcc lr, pc @ Make LR point into .entry.text so + @ that we will get a dump of the + @ exception stack for this frame. +#ifdef CONFIG_CC_IS_GCC + movcc ip, r0 @ Store the old SP in the frame record. + stmdbcc sp!, {fp, ip, lr, pc} @ Push frame record + addcc fp, sp, #12 +#else + stmdbcc sp!, {fp, lr} @ Push frame record + movcc fp, sp +#endif // CONFIG_CC_IS_GCC +#endif // CONFIG_UNWINDER_ARM + .endif +#endif // CONFIG_IRQSTACKS + bl generic_handle_arch_irq + +#ifdef CONFIG_IRQSTACKS +#ifdef CONFIG_UNWINDER_ARM + mov sp, fpreg @ Restore original SP +#else + mov fp, r8 @ Restore original FP + mov sp, r9 @ Restore original SP +#endif // CONFIG_UNWINDER_ARM +#endif // CONFIG_IRQSTACKS #else arch_irq_handler_default #endif @@ -199,7 +247,7 @@ ENDPROC(__dabt_svc) .align 5 __irq_svc: svc_entry - irq_handler + irq_handler from_user=0 #ifdef CONFIG_PREEMPTION ldr r8, [tsk, #TI_PREEMPT] @ get preempt count @@ -426,7 +474,7 @@ ENDPROC(__dabt_usr) __irq_usr: usr_entry kuser_cmpxchg_check - irq_handler + irq_handler from_user=1 get_thread_info tsk mov why, #0 b ret_to_user_from_irq diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index b79975bd988c..abb0aa679bba 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -43,6 +43,25 @@ unsigned long irq_err_count; +#ifdef CONFIG_IRQSTACKS + +asmlinkage DEFINE_PER_CPU_READ_MOSTLY(u8 *, irq_stack_ptr); + +static void __init init_irq_stacks(void) +{ + u8 *stack; + int cpu; + + for_each_possible_cpu(cpu) { + stack = (u8 *)__get_free_pages(GFP_KERNEL, THREAD_SIZE_ORDER); + if (WARN_ON(!stack)) + break; + per_cpu(irq_stack_ptr, cpu) = &stack[THREAD_SIZE]; + } +} + +#endif + int arch_show_interrupts(struct seq_file *p, int prec) { #ifdef CONFIG_FIQ @@ -101,6 +120,10 @@ void __init init_IRQ(void) { int ret; +#ifdef CONFIG_IRQSTACKS + init_irq_stacks(); +#endif + if (IS_ENABLED(CONFIG_OF) && !machine_desc->init_irq) irqchip_init(); else diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 89be21ec3b52..b42c446cec9a 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -66,6 +66,19 @@ void dump_backtrace_entry(unsigned long where, unsigned long from, { unsigned long end = frame + 4 + sizeof(struct pt_regs); + if (IS_ENABLED(CONFIG_UNWINDER_FRAME_POINTER) && + IS_ENABLED(CONFIG_CC_IS_GCC) && + end > ALIGN(frame, THREAD_SIZE)) { + /* + * If we are walking past the end of the stack, it may be due + * to the fact that we are on an IRQ or overflow stack. In this + * case, we can load the address of the other stack from the + * frame record. + */ + frame = ((unsigned long *)frame)[-2] - 4; + end = frame + 4 + sizeof(struct pt_regs); + } + #ifdef CONFIG_KALLSYMS printk("%s[<%08lx>] (%ps) from [<%08lx>] (%pS)\n", loglvl, where, (void *)where, from, (void *)from); @@ -278,7 +291,7 @@ static int __die(const char *str, int err, struct pt_regs *regs) if (!user_mode(regs) || in_interrupt()) { dump_mem(KERN_EMERG, "Stack: ", regs->ARM_sp, - THREAD_SIZE + (unsigned long)task_stack_page(tsk)); + ALIGN(regs->ARM_sp, THREAD_SIZE)); dump_backtrace(regs, tsk, KERN_EMERG); dump_instr(KERN_EMERG, regs); } diff --git a/arch/arm/lib/backtrace-clang.S b/arch/arm/lib/backtrace-clang.S index 5b4bca85d06d..290c52a60fc6 100644 --- a/arch/arm/lib/backtrace-clang.S +++ b/arch/arm/lib/backtrace-clang.S @@ -197,6 +197,14 @@ finished_setup: cmp sv_fp, frame @ next frame must be mov frame, sv_fp @ above the current frame +#ifdef CONFIG_IRQSTACKS + @ + @ Kernel stacks may be discontiguous in memory. If the next + @ frame is below the previous frame, accept it as long as it + @ lives in kernel memory. + @ + cmpls sv_fp, #PAGE_OFFSET +#endif bhi for_each_frame 1006: adr r0, .Lbad diff --git a/arch/arm/lib/backtrace.S b/arch/arm/lib/backtrace.S index e8408f22d4dc..293a2716bd20 100644 --- a/arch/arm/lib/backtrace.S +++ b/arch/arm/lib/backtrace.S @@ -98,6 +98,14 @@ for_each_frame: tst frame, mask @ Check for address exceptions cmp sv_fp, frame @ next frame must be mov frame, sv_fp @ above the current frame +#ifdef CONFIG_IRQSTACKS + @ + @ Kernel stacks may be discontiguous in memory. If the next + @ frame is below the previous frame, accept it as long as it + @ lives in kernel memory. + @ + cmpls sv_fp, #PAGE_OFFSET +#endif bhi for_each_frame 1006: adr r0, .Lbad -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel