From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5680DC433DB for ; Fri, 19 Mar 2021 22:04:57 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DC99461975 for ; Fri, 19 Mar 2021 22:04:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DC99461975 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:References: Cc:To:From:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=QupHbfVRtMqnZvBc3JBz8zthVQIRuAculyOg/1LXoNY=; b=lU6y+N502o5yJaVDaDaqSqR/t N5QF2Px+VJL6G58l2efq+FGXqFQOpv/gjZxFvEMYLHcrcwa/chUWbNX+JXTmhvRzm+axFPDqNVQ6l 0L+uQ0OO7useJSjut4INPL6v/xcv9hbKElNg6hVy/f1N0RF8eNVPslne+d0Mc+kmxSKc+ZU3yPIEP 1HvkkhdkN8Ru3Nra7fQh61rd6ngJquLDJAA19EUt8JiGfGUl7MqceP/poCy51XUYEou7tMFWuDr9t FY+GFuQbTGCCyJqQ/l7IFOgM+pWiFHvDa1GuvVKuTdL+SCVrCJDdW/ztp7Tx15RCyzwATkQ6qi8oS zTPDWKJTA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lNNDE-008EKL-HH; Fri, 19 Mar 2021 22:03:20 +0000 Received: from linux.microsoft.com ([13.77.154.182]) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lNND8-008EJq-H2 for linux-arm-kernel@lists.infradead.org; Fri, 19 Mar 2021 22:03:16 +0000 Received: from [192.168.254.32] (unknown [47.187.194.202]) by linux.microsoft.com (Postfix) with ESMTPSA id 17C3420B39C5; Fri, 19 Mar 2021 15:03:10 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 17C3420B39C5 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1616191390; bh=Ky/VX3m6s8WqSfAj51xXeUV0UTf0QhLUBID5a5TMekY=; h=Subject:From:To:Cc:References:Date:In-Reply-To:From; b=LZGMzILyYMHPo2MwcAJb2YdbYvUL0Usx/BQwXAmLIjsHEa3oDvUSGT3nVqOil03Yq aAnZUDgHGIQ/A4Pw3mow5BZKNp8tvgX/tPp/k5efVfXKKH4elb0ZAQonJJV3PjUcuu +MJbCwEd4dZuj/dxreGrP04TMHWs6UTAEb21IEOM= Subject: Re: [RFC PATCH v2 1/8] arm64: Implement stack trace termination record From: "Madhavan T. Venkataraman" To: Mark Brown Cc: mark.rutland@arm.com, jpoimboe@redhat.com, jthierry@redhat.com, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, live-patching@vger.kernel.org, linux-kernel@vger.kernel.org References: <5997dfe8d261a3a543667b83c902883c1e4bd270> <20210315165800.5948-1-madvenka@linux.microsoft.com> <20210315165800.5948-2-madvenka@linux.microsoft.com> <20210318150905.GL5469@sirena.org.uk> <8591e34a-c181-f3ff-e691-a6350225e5b4@linux.microsoft.com> <20210319123023.GC5619@sirena.org.uk> <5dbaf34f-b2fc-b9b8-3918-83356f2f752a@linux.microsoft.com> <6e3ac22b-99b8-7d99-59bd-6a2d1158b3c9@linux.microsoft.com> Message-ID: <254ed4a1-8342-d879-2fbc-3933118df949@linux.microsoft.com> Date: Fri, 19 Mar 2021 17:03:09 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <6e3ac22b-99b8-7d99-59bd-6a2d1158b3c9@linux.microsoft.com> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210319_220315_017779_73F52761 X-CRM114-Status: GOOD ( 33.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 3/19/21 1:19 PM, Madhavan T. Venkataraman wrote: > > > On 3/19/21 9:29 AM, Madhavan T. Venkataraman wrote: >> >> >> On 3/19/21 7:30 AM, Mark Brown wrote: >>> On Thu, Mar 18, 2021 at 03:26:13PM -0500, Madhavan T. Venkataraman wrote: >>>> On 3/18/21 10:09 AM, Mark Brown wrote: >>> >>>>> If we are going to add the extra record there would probably be less >>>>> potential for confusion if we pointed it at some sensibly named dummy >>>>> function so anything or anyone that does see it on the stack doesn't get >>>>> confused by a NULL. >>> >>>> I agree. I will think about this some more. If no other solution presents >>>> itself, I will add the dummy function. >>> >>> After discussing this with Mark Rutland offlist he convinced me that so >>> long as we ensure the kernel doesn't print the NULL record we're >>> probably OK here, the effort setting the function pointer up correctly >>> in all circumstances (especially when we're not in the normal memory >>> map) is probably not worth it for the limited impact it's likely to have >>> to see the NULL pointer (probably mainly a person working with some >>> external debugger). It should be noted in the changelog though, and/or >>> merged in with the relevant change to the unwinder. >>> >> >> OK. I will add a comment as well as note it in the changelog. >> >> Thanks to both of you. >> >> Madhavan >> > > I thought about this some more. I think I have a simple solution. I will > prepare a patch and send it out. If you and Mark Rutland approve, I will > include it in the next version of this RFC. > > Madhavan > I solved this by using existing functions logically instead of inventing a dummy function. I initialize pt_regs->stackframe[1] to an existing function so that the stack trace will not show a 0x0 entry as well as the kernel and gdb will show identical stack traces. For all task stack traces including the idle tasks, the stack trace will end at copy_thread() as copy_thread() is the function that initializes the pt_regs and the first stack frame for a task. For EL0 exceptions, the stack trace will end with vectors() as vectors entries call the EL0 handlers. Here are sample stack traces (I only show the ending of each trace): Idle task on primary CPU ======================== ... [ 0.022557] start_kernel+0x5b8/0x5f4 [ 0.022570] __primary_switched+0xa8/0xb8 [ 0.022578] copy_thread+0x0/0x188 Idle task on secondary CPU ========================== ... [ 0.023397] secondary_start_kernel+0x188/0x1e0 [ 0.023406] __secondary_switched+0x40/0x88 [ 0.023415] copy_thread+0x0/0x188 All other kernel threads ======================== ... [ 13.501062] ret_from_fork+0x10/0x18 [ 13.507998] copy_thread+0x0/0x188 User threads (EL0 exception) ============ write(2) system call example: ... [ 521.686148] vfs_write+0xc8/0x2c0 [ 521.686156] ksys_write+0x74/0x108 [ 521.686161] __arm64_sys_write+0x24/0x30 [ 521.686166] el0_svc_common.constprop.0+0x70/0x1a8 [ 521.686175] do_el0_svc+0x2c/0x98 [ 521.686180] el0_svc+0x2c/0x70 [ 521.686188] el0_sync_handler+0xb0/0xb8 [ 521.686193] el0_sync+0x17c/0x180 [ 521.686198] vectors+0x0/0x7d8 Here are the code changes: ======================================================================== diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index a31a0a713c85..514307e80b79 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -261,16 +261,19 @@ alternative_else_nop_endif stp lr, x21, [sp, #S_LR] /* - * For exceptions from EL0, terminate the callchain here. + * For exceptions from EL0, terminate the callchain here at + * task_pt_regs(current)->stackframe. + * * For exceptions from EL1, create a synthetic frame record so the * interrupted code shows up in the backtrace. */ .if \el == 0 - mov x29, xzr + ldr x17, =vectors + stp xzr, x17, [sp, #S_STACKFRAME] .else stp x29, x22, [sp, #S_STACKFRAME] - add x29, sp, #S_STACKFRAME .endif + add x29, sp, #S_STACKFRAME #ifdef CONFIG_ARM64_SW_TTBR0_PAN alternative_if_not ARM64_HAS_PAN diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 66b0e0b66e31..699e0dd313a1 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -393,6 +393,29 @@ SYM_FUNC_START_LOCAL(__create_page_tables) ret x28 SYM_FUNC_END(__create_page_tables) + /* + * The boot task becomes the idle task for the primary CPU. The + * CPU startup task on each secondary CPU becomes the idle task + * for the secondary CPU. + * + * The idle task does not require pt_regs. But create a dummy + * pt_regs so that task_pt_regs(idle_task)->stackframe can be + * set up to be the last frame on the idle task stack just like + * all the other kernel tasks. This helps the unwinder to + * terminate the stack trace at a well-known stack offset. + * + * Also, set up the last return PC to be copy_thread() just + * like all the other kernel tasks so that the stack trace of + * all kernel tasks ends with the same function. + */ + .macro setup_last_frame + sub sp, sp, #PT_REGS_SIZE + ldr x17, =copy_thread + stp xzr, x17, [sp, #S_STACKFRAME] + add x29, sp, #S_STACKFRAME + adr x30, #0 + .endm + /* * The following fragment of code is executed with the MMU enabled. * @@ -447,8 +470,7 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif bl switch_to_vhe // Prefer VHE if possible add sp, sp, #16 - mov x29, #0 - mov x30, #0 + setup_last_frame b start_kernel SYM_FUNC_END(__primary_switched) @@ -606,8 +628,7 @@ SYM_FUNC_START_LOCAL(__secondary_switched) cbz x2, __secondary_too_slow msr sp_el0, x2 scs_load x2, x3 - mov x29, #0 - mov x30, #0 + setup_last_frame #ifdef CONFIG_ARM64_PTR_AUTH ptrauth_keys_init_cpu x2, x3, x4, x5 diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 325c83b1a24d..9050699ff67c 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -437,6 +437,12 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start, } p->thread.cpu_context.pc = (unsigned long)ret_from_fork; p->thread.cpu_context.sp = (unsigned long)childregs; + /* + * For the benefit of the unwinder, set up childregs->stackframe + * as the last frame for the new task. + */ + p->thread.cpu_context.fp = (unsigned long)childregs->stackframe; + childregs->stackframe[1] = (unsigned long)copy_thread; ptrace_hw_copy_thread(p); ====================================================================== If you approve, the above will become RFC Patch v3 1/8 in the next version. Let me know. Also, I could introduce an extra frame in the EL1 exception stack trace that includes vectors so the stack trace would look like this (timer interrupt example): call_timer_fn run_timer_softirq __do_softirq irq_exit __handle_domain_irq gic_handle_irq el1_irq vectors This way, if the unwinder finds vectors, it knows that it is an exception frame. Madhavan _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel