Re: instrumentation, was Re: core dump analysis

Eero Tamminen
2023-04-12T23:30:34Z

Hi Michael,

On 11.4.2023 11.24, Michael Schmitz wrote:
> Am 11.04.2023 um 19:19 schrieb Geert Uytterhoeven:
>> On Tue, Apr 11, 2023 at 6:56 AM Michael Schmitz <schmitzmic@gmail.com>
>>> Am 11.04.2023 um 12:20 schrieb Finn Thain:
>>>> A better solution might be be to port the existing instrumentation like
>>>> ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder
>>>> how portable that stuff is.

Another possibility is just using manually added kernel tracepoints:
https://www.kernel.org/doc/html/latest/trace/tracepoints.html

"git grep TRACE_EVENT" did not given any hits for arch/m68k/ though.

(I don't think those need any particular architecture support. One just
adds them to locations where it makes sense to trace activity that is
important, but which does not happen too often for the very small
overhead from disabled static trace point to be significant.)

> Looking at a few arch implementations, I'm utterly confused. Wouldn't
> know where to start.

Good article: https://lwn.net/Articles/132196/

(Ignore jprobes stuff as that's deprecated.)

Because probe replaces instruction at the probe point with a breakpoint:
https://www.kernel.org/doc/html/latest/trace/kprobes.html#how-does-a-kprobe-work

It needs architecture specific code to decode and simulate the replaced
instruction (its side-effects) when it is later executed "out-of-line".

This also causes architecture specific limitations on where the probe
can be set. Architecture may not support simulating all possible
instructions, just the most common ones (e.g. ones typically used on
function entry and exit points).

Besides that, AFAIK plain kprobes support should not have much
architecture specific stuff.

(I have no experience with kernel code, I just remember some LWN
articles about kprobes during past 2 decades, and I was involved with a
tool that did something similar in user-space, before Linux got uprobes
support.)

As to what architecture to use as an example, I think it would be best
to ask tracing maintainers for advice, as there seems to be a lot of
code which has been copied from each other at different points in time,
that might be nowadays better as shared rather than under each arch...

These architectures support kprobes:
------------------------------------
$ grep " ok " Documentation/features/debug/kprobes/arch-support.txt
| arc: | ok |
| arm: | ok |
| arm64: | ok |
| csky: | ok |
| ia64: | ok |
| mips: | ok |
| parisc: | ok |
| powerpc: | ok |
| riscv: | ok |
| s390: | ok |
| sh: | ok |
| sparc: | ok |
| x86: | ok |
------------------------------------

But only these seem to have probing functionality isolated nicely to its
own directory:
------------------------------------
$ ls arch/*/kernel/probes/

arch/arm64/kernel/probes/:
decode-insn.c decode-insn.h kprobes.c kprobes_trampoline.S Makefile
simulate-insn.c simulate-insn.h uprobes.c

arch/csky/kernel/probes/:
decode-insn.c decode-insn.h ftrace.c kprobes.c kprobes_trampoline.S
Makefile simulate-insn.c simulate-insn.h uprobes.c

arch/riscv/kernel/probes/:
decode-insn.c decode-insn.h ftrace.c kprobes.c Makefile rethook.c
rethook.h rethook_trampoline.S simulate-insn.c simulate-insn.h uprobes.c
------------------------------------

As to testing whether kprobes work, that could be easier with ftrace
support also present:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/ftrace/README

- Eero

PS. Looking at the kernel features list, m68k arch seems to be missing a
lot of other features too: