From: takahiro.akashi@linaro.org (AKASHI Takahiro)
To: linux-arm-kernel@lists.infradead.org
Subject: arm64 function_graph tracer panic with CONFIG_DYNAMIC_FTRACE
Date: Fri, 13 Nov 2015 14:51:07 +0900 [thread overview]
Message-ID: <56457A4B.601@linaro.org> (raw)
In-Reply-To: <20151112104243.GF5627@e104818-lin.cambridge.arm.com>
Catalin,
(add lakml back to Cc)
On 11/13/2015 01:01 AM, Catalin Marinas wrote:
> On Fri, Nov 13, 2015 at 12:00:42AM +0900, AKASHI Takahiro wrote:
>> If possible, please try again with "select HAVE_FUNCTION_GRAPH_FP_TEST" added
>> to CONFIG_ARM64.
>> It will insert an additional check. I hope this will help.
>
> So it looks the the frame is off by some amount:
I was able to reproduce the problem on hikey, too.
> # echo function_graph > /sys/kernel/debug/tracing/current_tracer
> Bad frame pointer: expected ffffffc976b3bd50, received ffffffc976b3be60
The difference of fp's was 0x110(sizeof(pt_reg_t)=0x120).
It seems that an interrupt happened here (and we might have unwound one extra frame?).
> from func psci_cpu_suspend return to ffffffc00008d058
Always in the same place.
> ------------[ cut here ]------------
> WARNING: at /work/Linux/linux-2.6-aarch64/kernel/trace/trace_functions_graph.c:223
> Modules linked in:
>
> CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.3.0+ #584
> Hardware name: Juno (DT)
> task: ffffffc976b22f00 ti: ffffffc976b38000 task.ti: ffffffc976b38000
> PC is at ftrace_return_to_handler+0x11c/0x150
> LR is at ftrace_return_to_handler+0x11c/0x150
> pc : [<ffffffc000172034>] lr : [<ffffffc000172034>] pstate: 800001c5
> sp : ffffffc976b3bdd0
> x29: ffffffc976b3bdd0 x28: ffffffc0006c9000
> x27: ffffffc000a29f20 x26: ffffffc976b3bf70
> x25: 000000cc2c13e054 x24: 0000000000000004
> x23: ffffffc000aeb790 x22: 0000000000000002
> x21: ffffffc975503200 x20: ffffffc000aeb850
> x19: 0000000000000002 x18: ffffffc000b3a920
> x17: 0000000000000000 x16: 0000000000000000
> x15: ffffffc000b3a5bf x14: 3739636666666666
> x13: 6620646576696563 x12: 6572202c30356462
> x11: 3362363739636666 x10: 00000000000000fc
> x9 : ffffffc9768070c0 x8 : 0000000000001dd0
> x7 : ffffffc9760f4c80 x6 : ffffffc9760f4c00
> x5 : 00000000000084c0 x4 : 0000000000000001
> x3 : ffffffc976b3bc50 x2 : 0000000000000001
> x1 : 000000000000007f x0 : 000000000000007f
>
> ---[ end trace ac97c44ba7dabcce ]---
> Call trace:
> [<ffffffc000172034>] ftrace_return_to_handler+0x11c/0x150
> [<ffffffc0000924bc>] return_to_handler+0x1c/0x40
> [<ffffffc0000924a0>] return_to_handler+0x0/0x40
> [<ffffffc0000924a0>] return_to_handler+0x0/0x40
> [<ffffffc0000924a0>] return_to_handler+0x0/0x40
> [<ffffffc0000924a0>] return_to_handler+0x0/0x40
> [<ffffffc0000924a0>] return_to_handler+0x0/0x40
> [<ffffffc0000924a0>] return_to_handler+0x0/0x40
These call stacks were:
(psci_cpu_suspend)
psci_suspend_finisher
(cpu_suspend_enter?)
cpu_suspend
cpu_psci_cpu_suspend
arm_cpuidle_suspend
arm_enter_idle_state
cpuidle_enter_state
cpuidle_enter
call_cpuidle
(cpu_startup_entry?)
and if cpuidle is turned off(cpuidle.off=1), the panic never happens.
That is all what I know at this moment.
-Takahiro AKASHI
> [<ffffffc000090208>] secondary_start_kernel+0x130/0x158
> Unable to handle kernel NULL pointer dereference at virtual address 00000000
> pgd = ffffffc000b68000
> [00000000] *pgd=00000009f69ea003, *pud=00000009f69ea003, *pmd=00000009f69eb003, *pte=006000002c010707
> Internal error: Oops: 96000006 [#1] PREEMPT SMP
[snip]
On 11/12/2015 07:42 PM, Catalin Marinas wrote:
> Hi Takahiro,
>
> I was trying to assess what's working/not working in ftrace on arm64 and
> whether your recent patches are meant for 4.4 or can wait until 4.5.
> However, without any patches applied on a 4.3 kernel, enabling the
> function_graph tracer when CONFIG_DYNAMIC_FTRACE is enabled panics the
> kernel. Strangely, it does not happen if only a single CPU is on.
> However, with function_graph tracer on, onlining a second CPU leads to a
> similar panic.
>
> Below is the kernel panic with 2 CPUs online on a Juno board. There
> isn't much stack trace information for CPU 5 where the fault happened,
> the frame pointer (x29) seems corrupted as well. Any idea? Thanks.
>
> # echo function_graph > /sys/kernel/debug/tracing/current_tracer
> swapper/5[0]: undefined instruction: pc=ffffffc000a46698
> Code: 0009ccf8 ffffffc0 00000000 00000000 (00000030)
> Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP
> Modules linked in:
> CPU: 5 PID: 0 Comm: swapper/5 Not tainted 4.3.0 #574
> Hardware name: Juno (DT)
> task: ffffffc976b23ac0 ti: ffffffc976b3c000 task.ti: ffffffc976b3c000
> PC is at cpu_online_bits+0x0/0x8
> LR is at cpu_online_bits+0x0/0x8
> pc : [<ffffffc000a46698>] lr : [<ffffffc000a46698>] pstate: 600001c5
> sp : ffffffc976b3ff40
> x29: 0000000000000002 x28: ffffffc000100c0c
> x27: ffffffc000aeb790 x26: ffffffc9758d5800
> x25: 0000000000000002 x24: ffffffc0000924a0
> x23: ffffffc976b3ff40 x22: ffffffc00056841c
> x21: ffffffc000a25b80 x20: ffffffc000aeb790
> x19: ffffffc9758d5800 x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000
> x15: 0000000000000000 x14: 0000000000000000
> x13: 0000000000000000 x12: 0000000034d5d91d
> x11: 0000000000000000 x10: 0000000000001000
> x9 : ffffffc9768070c0 x8 : 00000000000007d8
> x7 : 0000001b280eb7c4 x6 : 0000000000000015
> x5 : 000000000000000a x4 : ffffffc0792a4990
> x3 : ffffffffffffffa1 x2 : ffffffc0792a4800
> x1 : ffffffc0000949e0 x0 : ffffffffffffffa1
>
> Process swapper/5 (pid: 0, stack limit = 0xffffffc976b3c020)
> Stack: (0xffffffc976b3ff40 to 0xffffffc976b40000)
> ff40: ffffffc976b3ff70 ffffffc0000924a0 ffffffc976b3c000 ffffffc000a46000
> ff60: ffffffc000b20000 0000001b27d716c0 ffffffc976b3ffd0 ffffffc000090208
> ff80: 0000000000000005 0000000000000e12 ffffffc000b306a0 0000000000000000
> ffa0: 0000000000000000 0000000000000000 0000000080b65000 0000000080b68000
> ffc0: ffffffc0000827f0 0000000000000000 0000000000000000 00000000800827dc
> ffe0: 0000000000000000 0000000000000000 068824085611a721 280b8a0b05808446
> Call trace:
> Code: 0009ccf8 ffffffc0 00000000 00000000 (00000030)
> ---[ end trace 702bb2ed35464601 ]---
> Kernel panic - not syncing: Attempted to kill the idle task!
> CPU4: stopping
> CPU: 4 PID: 0 Comm: swapper/4 Tainted: G D 4.3.0 #574
> Hardware name: Juno (DT)
> Call trace:
> [<ffffffc00008aaa8>] dump_backtrace+0x0/0x150
> [<ffffffc00008ac1c>] show_stack+0x24/0x30
> [<ffffffc0003b21a0>] dump_stack+0x90/0xd0
> [<ffffffc000090880>] handle_IPI+0x280/0x290
> [<ffffffc00008255c>] gic_handle_irq+0x94/0xb0
> Exception stack(0xffffffc976b3bd70 to 0xffffffc976b3be90)
> bd60: 0000001b2ed1ee8c ffffffc000aeb850
> bd80: ffffffc976b3bec0 ffffffc0005681dc 0000000080000145 ffffffc00009244c
> bda0: 0000000000000000 0000000000000001 ffffffc976b38000 0000000000000003
> bdc0: ffffffc976b38000 ffffffc000a282a8 ffffffc976b38000 ffffffc0000ecb28
> bde0: 00000000000006ed ffffffc9768070c0 00000000000007e0 ffffffc0006d04a8
> be00: 0000000000000001 0000000000005a0e 0000000000000001 ffffffc0006d04a8
> be20: 0000000000000000 0000000000000000 0000000000000001 0000001b2ed1ee8c
> be40: ffffffc000aeb850 ffffffc9758d5600 0000000000000000 ffffffc000aeb790
> be60: 0000000000000004 0000001b2e40a7ec ffffffc976b3bf70 ffffffc000a29f20
> be80: ffffffc0006c9000 ffffffc976b3bec0
> [<ffffffc0000855ac>] el1_irq+0x6c/0xe0
> [<ffffffc00056842c>] cpuidle_enter+0x34/0x48
> [<ffffffc000100c30>] call_cpuidle+0x48/0x80
> [<ffffffc000100e90>] cpu_startup_entry+0x228/0x308
> [<ffffffc000090208>] secondary_start_kernel+0x130/0x158
> ---[ end Kernel panic - not syncing: Attempted to kill the idle task!
>
next prev parent reply other threads:[~2015-11-13 5:51 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-12 10:42 arm64 function_graph tracer panic with CONFIG_DYNAMIC_FTRACE Catalin Marinas
2015-11-13 5:51 ` AKASHI Takahiro [this message]
2015-11-13 15:16 ` Jungseok Lee
2015-11-16 1:56 ` AKASHI Takahiro
2015-11-16 13:45 ` Catalin Marinas
2015-11-16 15:48 ` Lorenzo Pieralisi
2015-11-17 0:39 ` AKASHI Takahiro
2015-11-17 10:12 ` Lorenzo Pieralisi
2015-11-16 14:20 ` Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56457A4B.601@linaro.org \
--to=takahiro.akashi@linaro.org \
--cc=linux-arm-kernel@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.