From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E5D2C433F5 for ; Tue, 21 Sep 2021 09:58:20 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E227760F4B for ; Tue, 21 Sep 2021 09:58:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E227760F4B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ft/gYPumbsbqPhEKWz0JzMksCcZTxOnZ1eQIDxxKPDQ=; b=39BO9RK1W74QID n+pKdKApwhei+WoCjXzeDTfsC7wDWjFNf9M+ik9pHN/aURfV/5llgVhTlqV477MEhCwVgxNe2mF7s O0vrI5dQJBihO7n3GgtmsTn5Tg4g6Hb+klyl9ebYpPTACq1wMcwE8BSVpk6/lKmRm1ElcWGCrStxN BO6X1TnlYIX8aR7SKznJrdhSffdYL4artMiAWoBpLaUaCS48aZ2+aPKaRFGLGkmGfSaNYEK4lEY4O CXQcp/u02hMLTkzAPqwUXuwpju1omV1NjYEQ2axY6GwQ8KnemjhVRc1qFWYRVo118Dqt8WpV8dPPq Xjt5SMP7r7HfPjPCxldg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mScVX-0045lP-KY; Tue, 21 Sep 2021 09:56:12 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mScTx-00454g-Gf for linux-arm-kernel@lists.infradead.org; Tue, 21 Sep 2021 09:54:35 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 791D061100; Tue, 21 Sep 2021 09:54:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1632218073; bh=0OG47JnBTM2nenK7KHc/R2SudD+WKXdm5qtzt0k7Dw4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bbPCFTfP6EffOGrMrZ7qlALRLcE0oqqcxKSY4WerQNVYfN1NGoIW5OLEmDZ50yjtN QfexhtikPoFbMBzWJJZeZZPSanQin4g7RYu4h41cx9Sow5JaY1bt32hO6+BkpsoVZu WISw5zqcxB6Woy2yZpWg3RrbPw4PjwnCkUTh5/9UCn46Oqg8yYHJJIM2c7+cAvppRK 5JmTHr89vMq56bsnGVAFLxe7s7RGlv510mOhDFkFn2uXuBacuviYMZr+dd+fO/kjIm tVY4UCfXAf1R3eXRMF0juPi20oh/lO3+fGZbG7diQRFKkv9eo9ejxsH0lIDxsuDWN2 zs6DFjbo13DVQ== From: Ard Biesheuvel To: linux@armlinux.org.uk Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij Subject: [PATCH 08/10] ARM: implement IRQ stacks Date: Tue, 21 Sep 2021 11:54:06 +0200 Message-Id: <20210921095408.133210-9-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210921095408.133210-1-ardb@kernel.org> References: <20210921095408.133210-1-ardb@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210921_025433_627766_F3BA493C X-CRM114-Status: GOOD ( 22.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well. Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet. Signed-off-by: Ard Biesheuvel --- arch/arm/Kconfig | 4 ++ arch/arm/kernel/entry-armv.S | 40 ++++++++++++++++++-- arch/arm/kernel/irq.c | 23 +++++++++++ arch/arm/kernel/traps.c | 15 +++++++- arch/arm/lib/backtrace-clang.S | 8 ++++ arch/arm/lib/backtrace.S | 8 ++++ 6 files changed, 94 insertions(+), 4 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 4f61c9789e7f..a8c1db0736f3 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1163,6 +1163,10 @@ config CURRENT_POINTER_IN_TPIDRURO def_bool y depends on SMP && CPU_32v6K && !CPU_V6 +config IRQSTACKS + def_bool y + depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK + config ARM_CPU_TOPOLOGY bool "Support cpu topology definition" depends on SMP && CPU_V7 diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 21896f702447..c1b1b125f647 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -36,11 +36,45 @@ /* * Interrupt handling. */ - .macro irq_handler + .macro irq_handler, from_user:req #ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER mov_l r1, handle_arch_irq mov r0, sp +#ifdef CONFIG_IRQSTACKS + mov_l r2, irq_stack_ptr @ Take base address + mrc p15, 0, r3, c13, c0, 4 @ Get CPU offset + mov r7, sp @ Preserve original SP + ldr sp, [r2, r3] @ Load SP from per-CPU var + .if \from_user == 0 + @ + @ If we took the interrupt while running in the kernel, we may already + @ be using the IRQ stack, so revert to the original value in that case. + @ + subs r3, sp, r7 @ SP above bottom of IRQ stack? + rsbscs r3, r3, #THREAD_SIZE @ ... and below the top? + movcs sp, r7 @ If so, revert to incoming SP + + @ + @ Inform the unwinder where the next frame lives + @ +#if defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_GCC) + movcc ip, r7 @ Store the old SP in the frame record. + movcc lr, pc @ Make LR point into .entry.text so + @ that we will get a dump of the + @ exception stack for this frame. + stmdbcc sp!, {fp, ip, lr, pc} + addcc fp, sp, #12 +#elif defined(CONFIG_UNWINDER_ARM) +ARM( mov fp, r7 ) +ARM( .setfp fp, sp ) +THUMB( .setfp r7, sp ) +#endif + .endif +#endif bl_m [r1] +#ifdef CONFIG_IRQSTACKS + mov sp, r7 @ Restore original SP +#endif #else arch_irq_handler_default #endif @@ -200,7 +234,7 @@ ENDPROC(__dabt_svc) .align 5 __irq_svc: svc_entry - irq_handler + irq_handler from_user=0 #ifdef CONFIG_PREEMPTION ldr r8, [tsk, #TI_PREEMPT] @ get preempt count @@ -427,7 +461,7 @@ ENDPROC(__dabt_usr) __irq_usr: usr_entry kuser_cmpxchg_check - irq_handler + irq_handler from_user=1 get_thread_info tsk mov why, #0 b ret_to_user_from_irq diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index 20ab1e607522..58af2adb1583 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -43,6 +43,25 @@ unsigned long irq_err_count; +#ifdef CONFIG_IRQSTACKS + +asmlinkage DEFINE_PER_CPU_READ_MOSTLY(u8 *, irq_stack_ptr); + +static void __init init_irq_stacks(void) +{ + u8 *stack; + int cpu; + + for_each_possible_cpu(cpu) { + stack = (u8 *)__get_free_pages(GFP_KERNEL, THREAD_SIZE_ORDER); + if (WARN_ON(!stack)) + break; + per_cpu(irq_stack_ptr, cpu) = &stack[THREAD_SIZE]; + } +} + +#endif + int arch_show_interrupts(struct seq_file *p, int prec) { #ifdef CONFIG_FIQ @@ -99,6 +118,10 @@ void __init init_IRQ(void) { int ret; +#ifdef CONFIG_IRQSTACKS + init_irq_stacks(); +#endif + if (IS_ENABLED(CONFIG_OF) && !machine_desc->init_irq) irqchip_init(); else diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 89be21ec3b52..b42c446cec9a 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -66,6 +66,19 @@ void dump_backtrace_entry(unsigned long where, unsigned long from, { unsigned long end = frame + 4 + sizeof(struct pt_regs); + if (IS_ENABLED(CONFIG_UNWINDER_FRAME_POINTER) && + IS_ENABLED(CONFIG_CC_IS_GCC) && + end > ALIGN(frame, THREAD_SIZE)) { + /* + * If we are walking past the end of the stack, it may be due + * to the fact that we are on an IRQ or overflow stack. In this + * case, we can load the address of the other stack from the + * frame record. + */ + frame = ((unsigned long *)frame)[-2] - 4; + end = frame + 4 + sizeof(struct pt_regs); + } + #ifdef CONFIG_KALLSYMS printk("%s[<%08lx>] (%ps) from [<%08lx>] (%pS)\n", loglvl, where, (void *)where, from, (void *)from); @@ -278,7 +291,7 @@ static int __die(const char *str, int err, struct pt_regs *regs) if (!user_mode(regs) || in_interrupt()) { dump_mem(KERN_EMERG, "Stack: ", regs->ARM_sp, - THREAD_SIZE + (unsigned long)task_stack_page(tsk)); + ALIGN(regs->ARM_sp, THREAD_SIZE)); dump_backtrace(regs, tsk, KERN_EMERG); dump_instr(KERN_EMERG, regs); } diff --git a/arch/arm/lib/backtrace-clang.S b/arch/arm/lib/backtrace-clang.S index 5b2cdb1003e3..9fefb391bb39 100644 --- a/arch/arm/lib/backtrace-clang.S +++ b/arch/arm/lib/backtrace-clang.S @@ -197,6 +197,14 @@ finished_setup: cmp sv_fp, frame @ next frame must be mov frame, sv_fp @ above the current frame +#ifdef CONFIG_IRQSTACKS + @ + @ Kernel stacks may be discontiguous in memory. If the next + @ frame is below the previous frame, accept it as long as it + @ lives in kernel memory. + @ + cmpls sv_fp, #PAGE_OFFSET +#endif bhi for_each_frame 1006: adr r0, .Lbad diff --git a/arch/arm/lib/backtrace.S b/arch/arm/lib/backtrace.S index e8408f22d4dc..293a2716bd20 100644 --- a/arch/arm/lib/backtrace.S +++ b/arch/arm/lib/backtrace.S @@ -98,6 +98,14 @@ for_each_frame: tst frame, mask @ Check for address exceptions cmp sv_fp, frame @ next frame must be mov frame, sv_fp @ above the current frame +#ifdef CONFIG_IRQSTACKS + @ + @ Kernel stacks may be discontiguous in memory. If the next + @ frame is below the previous frame, accept it as long as it + @ lives in kernel memory. + @ + cmpls sv_fp, #PAGE_OFFSET +#endif bhi for_each_frame 1006: adr r0, .Lbad -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel