From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E93B4C433E6 for ; Fri, 22 Jan 2021 13:37:16 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AEA6E235E4 for ; Fri, 22 Jan 2021 13:37:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AEA6E235E4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Mime-Version:References:In-Reply-To:Message-Id: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=HOKbsYjNK9zN8ut2Cqrsuvltjm4+PBSodDMNTKKqo3Q=; b=RCcYGnJaSZP7l1aysjx5AsI9X NmPCqru46mi8xPQV+21A3m+r0CTHVB5q9KcfNM9QPU66UarUnbeW2uC3yXWXbBdlMy/1Qa1MRMh0s I0cHV+D0DKvX0EbBJj7zHb6yvur2E3VLWOt7t2rrlucNEk909sxQG1+FPb6RATfWhQHV8ZUb7fu8p lS4YAVaVXnttC2cALkRrXIPtwSvrEjNyzkfqDose5vlYksE80r6WAMowaI9gcQc1vpUDIJSj5cI4b VYmsy6M03jt5lBXAc0JEGhSc8P1ofH37mfy+aAzEUf114Xy4N3KkWYrpg9RmE9WKbqm2t7WM5y6m4 paiAV5R3g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l2wbh-0006vk-0D; Fri, 22 Jan 2021 13:36:09 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l2wbd-0006v1-PA for linux-arm-kernel@lists.infradead.org; Fri, 22 Jan 2021 13:36:06 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 0CA1E235E4; Fri, 22 Jan 2021 13:36:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1611322564; bh=IZn22MWrfTYUq6thMEBa0y6cRsXftVtNVd0e+I1PtWw=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=lJmU4UDCueXcDKYIb85GHaeavxfmhYsD5AT4vDyrCaFaKoKlLtkyke0zJ8CGNia8l vZFt9qQTP8X/yCfR8Y9dTRnG+rdDbYdJtezOH7BYIwH716PwLxk5R7r6ej2SmlrJSv +urU8FjZ2tIiLI9twfb7SeoboK3B5Krti2eBz9NHhciuSQyILfYKhpiuK0kuKXPiNS xQF1kLL8x0GfsO8TA11UPa5mpHyRFL3x2r8zxqjKMMooXoSFAnoXuJ0HPzELOeyhYT OLQCJWA9psSatRUGX7co2XNrqtjxCh0g7LIVbebE7aIPa19mUh/Wrbi4B2vgyLpIGj Ku2/QgRBcAMrA== Date: Fri, 22 Jan 2021 22:36:01 +0900 From: Masami Hiramatsu To: Qais Yousef Subject: Re: [PATCH] arm64: kprobes: Fix Uexpected kernel BRK exception at EL1 Message-Id: <20210122223601.4b3b7e01c6ec583c0439f1c9@kernel.org> In-Reply-To: <20210122110909.3324607-1-qais.yousef@arm.com> References: <20210122110909.3324607-1-qais.yousef@arm.com> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; x86_64-pc-linux-gnu) Mime-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210122_083605_940265_49F9C929 X-CRM114-Status: GOOD ( 25.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jean-Philippe Brucker , Catalin Marinas , linux-kernel@vger.kernel.org, Masami Hiramatsu , Will Deacon , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, 22 Jan 2021 11:09:09 +0000 Qais Yousef wrote: > I was hitting the below panic continuously when attaching kprobes to > scheduler functions > > [ 159.045212] Unexpected kernel BRK exception at EL1 > [ 159.053753] Internal error: BRK handler: f2000006 [#1] PREEMPT SMP > [ 159.059954] Modules linked in: > [ 159.063025] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.11.0-rc4-00008-g1e2a199f6ccd #56 > [rt-app] [1] Exiting.[ 159.071166] Hardware name: ARM Juno development board (r2) (DT) > [ 159.079689] pstate: 600003c5 (nZCv DAIF -PAN -UAO -TCO BTYPE=--) > > [ 159.085723] pc : 0xffff80001624501c > [ 159.089377] lr : attach_entity_load_avg+0x2ac/0x350 > [ 159.094271] sp : ffff80001622b640 > [rt-app] [0] Exiting.[ 159.097591] x29: ffff80001622b640 x28: 0000000000000001 > [ 159.105515] x27: 0000000000000049 x26: ffff000800b79980 > > [ 159.110847] x25: ffff00097ef37840 x24: 0000000000000000 > [ 159.116331] x23: 00000024eacec1ec x22: ffff00097ef12b90 > [ 159.121663] x21: ffff00097ef37700 x20: ffff800010119170 > [rt-app] [11] Exiting.[ 159.126995] x19: ffff00097ef37840 x18: 000000000000000e > [ 159.135003] x17: 0000000000000001 x16: 0000000000000019 > [ 159.140335] x15: 0000000000000000 x14: 0000000000000000 > [ 159.145666] x13: 0000000000000002 x12: 0000000000000002 > [ 159.150996] x11: ffff80001592f9f0 x10: 0000000000000060 > [ 159.156327] x9 : ffff8000100f6f9c x8 : be618290de0999a1 > [ 159.161659] x7 : ffff80096a4b1000 x6 : 0000000000000000 > [ 159.166990] x5 : ffff00097ef37840 x4 : 0000000000000000 > [ 159.172321] x3 : ffff000800328948 x2 : 0000000000000000 > [ 159.177652] x1 : 0000002507d52fec x0 : ffff00097ef12b90 > [ 159.182983] Call trace: > [ 159.185433] 0xffff80001624501c > [ 159.188581] update_load_avg+0x2d0/0x778 > [ 159.192516] enqueue_task_fair+0x134/0xe20 > [ 159.196625] enqueue_task+0x4c/0x2c8 > [ 159.200211] ttwu_do_activate+0x70/0x138 > [ 159.204147] sched_ttwu_pending+0xbc/0x160 > [ 159.208253] flush_smp_call_function_queue+0x16c/0x320 > [ 159.213408] generic_smp_call_function_single_interrupt+0x1c/0x28 > [ 159.219521] ipi_handler+0x1e8/0x3c8 > [ 159.223106] handle_percpu_devid_irq+0xd8/0x460 > [ 159.227650] generic_handle_irq+0x38/0x50 > [ 159.231672] __handle_domain_irq+0x6c/0xc8 > [ 159.235781] gic_handle_irq+0xcc/0xf0 > [ 159.239452] el1_irq+0xb4/0x180 > [ 159.242600] rcu_is_watching+0x28/0x70 > [ 159.246359] rcu_read_lock_held_common+0x44/0x88 > [ 159.250991] rcu_read_lock_any_held+0x30/0xc0 > [ 159.255360] kretprobe_dispatcher+0xc4/0xf0 > [ 159.259555] __kretprobe_trampoline_handler+0xc0/0x150 > [ 159.264710] trampoline_probe_handler+0x38/0x58 > [ 159.269255] kretprobe_trampoline+0x70/0xc4 > [ 159.273450] run_rebalance_domains+0x54/0x80 > [ 159.277734] __do_softirq+0x164/0x684 > [ 159.281406] irq_exit+0x198/0x1b8 > [ 159.284731] __handle_domain_irq+0x70/0xc8 > [ 159.288840] gic_handle_irq+0xb0/0xf0 > [ 159.292510] el1_irq+0xb4/0x180 > [ 159.295658] arch_cpu_idle+0x18/0x28 > [ 159.299245] default_idle_call+0x9c/0x3e8 > [ 159.303265] do_idle+0x25c/0x2a8 > [ 159.306502] cpu_startup_entry+0x2c/0x78 > [ 159.310436] secondary_start_kernel+0x160/0x198 > [ 159.314984] Code: d42000c0 aa1e03e9 d42000c0 aa1e03e9 (d42000c0) > > After a bit of head scratching and debugging it turned out that it is > due to kprobe handler being interrupted by a tick that causes us to go > into (I think another) kprobe handler. > > The culprit was kprobe_breakpoint_ss_handler() returning DBG_HOOK_ERROR > which leads to the Unexpected kernel BRK exception. > > Reverting commit ba090f9cafd5 ("arm64: kprobes: Remove redundant > kprobe_step_ctx") seemed to fix the problem for me. > > Further analysis showed that kcb->kprobe_status is set to > KPROBE_REENTER when the error occurs. By teaching > kprobe_breakpoint_ss_handler() to handle this status I can no longer > reproduce the problem. Very good catch! Yes, this missed the reentered kprobe case. Acked-by: Masami Hiramatsu > > Fixes: ba090f9cafd5 ("arm64: kprobes: Remove redundant kprobe_step_ctx") > Signed-off-by: Qais Yousef > --- > > Another change in behavior I noticed is that before ba090f9cafd5 ("arm64: > kprobes: Remove redundant kprobe_step_ctx") if 'cur' was NULL we wouldn't > return DBG_HOOK_ERROR, but after the change we do. It should not happen, since the KPROBES_BRK_SS_IMM must be used only for kprobes's second break which must happen on the trampoline instruction buffer, which must set current kprobes before execution. Thank you, > > I didn't hit a problem because of that it's just something I noticed when > I realized that this commit was causing my problem. > > > arch/arm64/kernel/probes/kprobes.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c > index 89c64ada8732..66aac2881ba8 100644 > --- a/arch/arm64/kernel/probes/kprobes.c > +++ b/arch/arm64/kernel/probes/kprobes.c > @@ -352,8 +352,8 @@ kprobe_breakpoint_ss_handler(struct pt_regs *regs, unsigned int esr) > unsigned long addr = instruction_pointer(regs); > struct kprobe *cur = kprobe_running(); > > - if (cur && (kcb->kprobe_status == KPROBE_HIT_SS) > - && ((unsigned long)&cur->ainsn.api.insn[1] == addr)) { > + if (cur && (kcb->kprobe_status & (KPROBE_HIT_SS | KPROBE_REENTER)) && > + ((unsigned long)&cur->ainsn.api.insn[1] == addr)) { > kprobes_restore_local_irqflag(kcb, regs); > post_kprobe_handler(cur, kcb, regs); > > -- > 2.25.1 > -- Masami Hiramatsu _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel