Problem with ftrace uprobes on ARM

Problem with ftrace uprobes on ARM

[Christian Ascheberg]

Hello,

I want to trace a program using ftrace uprobes on an Altera Cyclone V 
SoC board which has a ARMv7 processor, but I cannot get it to work. 
Every source I read says that uprobes should be supported on ARM.
So I hope that this is the right place to ask this question. I tried 
different kernel versions from Altera and github.com/torvalds/linux/, 
including 4.0, 4.1 and the latest 4.2-RC6.
The Linux config is the default socfpga_defconfig plus 
CONFIG_UPROBE_EVENT=y. Can somebody tell me if support should be 
available as I expect? These are the commands that I tried:

# objdump -t benchmark | grep "calc_sum"
00008574 g     F .text    00000020              calc_sum

# echo 'p /home/root/benchmark:0x8574' > 
/sys/kernel/debug/tracing/uprobe_events

# echo 1 > 
/sys/kernel/debug/tracing/events/uprobes/p_benchmark_0x8574/enable
-sh: echo: write error: Invalid argument

So the probe point is registered, but I cannot enable it. dmesg tells me 
the same without providing any more details.
I tried to do the same using Perf but it crashed my program, presumably 
when hitting the probe point.

Thanks
Christian

Problem with ftrace uprobes on ARM

[Rabin Vincent]

On Mon, Aug 10, 2015 at 07:49:11PM +0200, Christian Ascheberg wrote:
> # objdump -t benchmark | grep "calc_sum"
> 00008574 g     F .text    00000020              calc_sum
> 
> # echo 'p /home/root/benchmark:0x8574' >
> /sys/kernel/debug/tracing/uprobe_events
> 
> # echo 1 >
> /sys/kernel/debug/tracing/events/uprobes/p_benchmark_0x8574/enable
> -sh: echo: write error: Invalid argument

The offset of the instruction in the file should be used, not the virtual
address.  For example, in the following example ls_main() is at 0x0009aec0 but
to set the breakpoint one actually has to use 0x92ec0:

$ arm-none-linux-gnueabi-nm busybox_unstripped | grep ls_main
0009aec0 T ls_main

$ arm-none-linux-gnueabi-objdump -F -d busybox_unstripped | grep '<ls_main>'
0009aec0 <ls_main> (File Offset: 0x92ec0):

By the way, I hope "benchmark" does not use Thumb instructions, because
uprobes on those aren't supported.  See arch_uprobe_analyze_insn().

Problem with ftrace uprobes on ARM

[Christian Ascheberg]

Thanks, your answer was really helpful. I did not know that Thumb 
instructions were not supported, which solved my initial problem when 
using Perf as well.

Is there any documentation regarding limitations of perf/ftrace support 
on ARM (compared to x86)?

For example, I also noticed that 'perf help' lists 'trace' command, but 
running 'perf trace' says 'is not a perf-command'.

Problem with ftrace uprobes on ARM

[Rabin Vincent]

On Wed, Aug 12, 2015 at 11:27:35AM +0200, Christian Ascheberg wrote:
> Is there any documentation regarding limitations of perf/ftrace support on
> ARM (compared to x86)?

There's Documentation/features/ which lists the status of architecture
support for various kernel options.  I don't know of any other
documentation.

> For example, I also noticed that 'perf help' lists 'trace' command, but
> running 'perf trace' says 'is not a perf-command'.

That doesn't have anything to do with something missing in ARM afaics.
It's probably because your perf wasn't built with libaudit.  Information
is printed about this when you build perf.  The fact that 'perf help'
lists unavailable commands is probably a bug.

 ~/dev/linux/tools/perf$ make
   BUILD:   Doing 'make -j8' parallel build
 config/Makefile:366: No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev