Re: instrumentation, was Re: core dump analysis

Eero Tamminen
2023-04-15T15:50:18Z

Hi,

On 13.4.2023 2.22, Eero Tamminen wrote:
> On 11.4.2023 11.24, Michael Schmitz wrote:
>> Am 11.04.2023 um 19:19 schrieb Geert Uytterhoeven:
>>> On Tue, Apr 11, 2023 at 6:56 AM Michael Schmitz <schmitzmic@gmail.com>
>>>> Am 11.04.2023 um 12:20 schrieb Finn Thain:
>>>>> A better solution might be be to port the existing instrumentation
>>>>> like
>>>>> ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder
>>>>> how portable that stuff is.
>
> Another possibility is just using manually added kernel tracepoints:
> https://www.kernel.org/doc/html/latest/trace/tracepoints.html
>
> "git grep TRACE_EVENT" did not given any hits for arch/m68k/ though.
>
> (I don't think those need any particular architecture support. One just
> adds them to locations where it makes sense to trace activity that is
> important, but which does not happen too often for the very small
> overhead from disabled static trace point to be significant.)
>
>
>> Looking at a few arch implementations, I'm utterly confused. Wouldn't
>> know where to start.
>
> Good article: https://lwn.net/Articles/132196/
>
> (Ignore jprobes stuff as that's deprecated.)
>
>
> Because probe replaces instruction at the probe point with a breakpoint:
> https://www.kernel.org/doc/html/latest/trace/kprobes.html#how-does-a-kprobe-work
>
> It needs architecture specific code to decode and simulate the replaced
> instruction (its side-effects) when it is later executed "out-of-line".
>
> This also causes architecture specific limitations on where the probe
> can be set.  Architecture may not support simulating all possible
> instructions, just the most common ones (e.g. ones typically used on
> function entry and exit points).

These are the kernel symbol entry point instructions that get executed
when booting Atari Falcon from IDE (using emulator LILO) to Busybox init:
4524 link.w
4 movea.l
3 move.l
3 clr.l
2 movem.l
2 dc.w
2 btst.b
1 subq.l
1 ori.w
1 move.w
1 lea.l
1 jsr
1 bra.b

For Linus' 6.2 kernel built with this config:
https://git.tuxfamily.org/hatari/hatari.git/tree/tools/linux/kernel.config

Using "gcc-m68k-linux-gnu" (4:10.2.1-1) in Debian Stable.

Count is for how many of the executed kernel symbol addresses had given
instruction.

I'm a bit surprised how many functions (symbol addresses) start with
link.w instruction, instead of e.g. move* instruction.

(dc.w is check for emulator NatFeats API.)

- Eero

PS. Above info is produced by building & booting Linux under Hatari
emulator as documented here:
https://hatari.tuxfamily.org/doc/m68k-linux.txt

And using:
-------------------
hatari ... --parse sym-breakpoint.ini > instr.txt
-------------------

Where "sym-breakpoint.ini" is:
-------------------
b PConSymbol > 0 :trace :quiet :file pc-disasm.ini
-------------------

And "pc-disasm.ini" file is:
-------------------
d pc-"pc+2"
-------------------

Then to get counts from the output:
-------------------
$ cat instr.txt | grep -E -e '^\$' | sort -u |\
cut -b46-52 | sort | uniq -c | sort -nr
-------------------

(Match disassembly lines, drop duplicate function calls, count uniq
entry point instructions.)

> Besides that, AFAIK plain kprobes support should not have much
> architecture specific stuff.
>
>
> (I have no experience with kernel code, I just remember some LWN
> articles about kprobes during past 2 decades, and I was involved with a
> tool that did something similar in user-space, before Linux got uprobes
> support.)
>
>
>
> As to what architecture to use as an example, I think it would be best
> to ask tracing maintainers for advice, as there seems to be a lot of
> code which has been copied from each other at different points in time,
> that might be nowadays better as shared rather than under each arch...
>
> These architectures support kprobes:
> ------------------------------------
> $ grep " ok " Documentation/features/debug/kprobes/arch-support.txt
>     |         arc: |  ok  |
>     |         arm: |  ok  |
>     |       arm64: |  ok  |
>     |        csky: |  ok  |
>     |        ia64: |  ok  |
>     |        mips: |  ok  |
>     |      parisc: |  ok  |
>     |     powerpc: |  ok  |
>     |       riscv: |  ok  |
>     |        s390: |  ok  |
>     |          sh: |  ok  |
>     |       sparc: |  ok  |
>     |         x86: |  ok  |
> ------------------------------------
>
> But only these seem to have probing functionality isolated nicely to its
> own directory:
> ------------------------------------
> $ ls arch/*/kernel/probes/
>
> arch/arm64/kernel/probes/:
> decode-insn.c  decode-insn.h  kprobes.c  kprobes_trampoline.S  Makefile
> simulate-insn.c  simulate-insn.h  uprobes.c
>
> arch/csky/kernel/probes/:
> decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  kprobes_trampoline.S
> Makefile  simulate-insn.c  simulate-insn.h  uprobes.c
>
> arch/riscv/kernel/probes/:
> decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  Makefile  rethook.c
> rethook.h  rethook_trampoline.S  simulate-insn.c  simulate-insn.h
> uprobes.c
> ------------------------------------
>
>
> As to testing whether kprobes work, that could be easier with ftrace
> support also present:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/ftrace/README
>
>
>     - Eero
>
> PS. Looking at the kernel features list, m68k arch seems to be missing a
> lot of other features too:
> https://www.kernel.org/doc/html/latest/m68k/features.html
>