Re: [RFC PATCH v3 00/11] powerpc: Add support for ftrace direct and BPF trampolines

[Vishal Chourasia <vishalc@linux.ibm.com>]

On Fri, Jun 21, 2024 at 12:24:03AM +0530, Naveen N Rao wrote:
> This is v3 of the patches posted here:
> http://lkml.kernel.org/r/cover.1718008093.git.naveen@kernel.org
> 
> Since v2, I have addressed review comments from Steven and Masahiro 
> along with a few fixes. Patches 7-11 are new in this series and add 
> support for ftrace direct and bpf trampolines. 
> 
> This series depends on the patch series from Benjamin Gray adding 
> support for patch_ulong():
> http://lkml.kernel.org/r/20240515024445.236364-1-bgray@linux.ibm.com
> 
> 
> - Naveen

Hello Naveen,

I've noticed an issue with `kstack()` in bpftrace [1] when using `kfunc` 
compared to `kprobe`. Despite trying all three modes specified in the 
documentation (bpftrace, perf, or raw), the stack isn't unwinding 
properly with `kfunc`. 

[1] https://github.com/bpftrace/bpftrace/blob/master/man/adoc/bpftrace.adoc#kstack


for mode in modes; do
	run bpftrace with kfunc
	disable cpu
	kill bpftrace
	run bpftrace with kprobe
	enable cpu
	kill bpftrace

# ./kprobe_vs_kfunc.sh
+ bpftrace -e 'kfunc:vmlinux:build_sched_domains {@[kstack(bpftrace), comm, tid]=count();}'
Attaching 1 probe...
+ chcpu -d 2-3
CPU 2 disabled
CPU 3 disabled
+ kill 35214

@[
    bpf_prog_cfd8d6c8bb4898ce+972
, cpuhp/2, 33]: 1
@[
    bpf_prog_cfd8d6c8bb4898ce+972
, cpuhp/3, 38]: 1

+ bpftrace -e 'kprobe:build_sched_domains {@[kstack(bpftrace), comm, tid]=count();}'
Attaching 1 probe...
+ chcpu -e 2-3
CPU 2 enabled
CPU 3 enabled
+ kill 35221

@[
    0x80000007642bdfb4
    partition_sched_domains_locked+1304
    rebuild_sched_domains_locked+216
    cpuset_handle_hotplug+1148
    sched_cpu_activate+664
    cpuhp_invoke_callback+480
    cpuhp_thread_fun+244
    smpboot_thread_fn+460
    kthread+308
    start_kernel_thread+20
, cpuhp/3, 38]: 1
@[
    0x80000007524b34a4
    partition_sched_domains_locked+1304
    rebuild_sched_domains_locked+216
    cpuset_handle_hotplug+1148
    sched_cpu_activate+664
    cpuhp_invoke_callback+480
    cpuhp_thread_fun+244
    smpboot_thread_fn+460
    kthread+308
    start_kernel_thread+20
, cpuhp/2, 33]: 1

+ bpftrace -e 'kfunc:vmlinux:build_sched_domains {@[kstack(perf), comm, tid]=count();}'
Attaching 1 probe...
+ chcpu -d 2-3
CPU 2 disabled
CPU 3 disabled
+ kill 35229

@[
        c008000003433454 bpf_prog_cfd8d6c8bb4898ce+960
, cpuhp/3, 38]: 1
@[
        c008000003433454 bpf_prog_cfd8d6c8bb4898ce+960
, cpuhp/2, 33]: 1


+ bpftrace -e 'kprobe:build_sched_domains {@[kstack(perf), comm, tid]=count();}'
Attaching 1 probe...
+ chcpu -e 2-3
CPU 2 enabled
CPU 3 enabled
+ kill 35235

@[
        80000007524b379c 0x80000007524b379c
        c000000000206268 partition_sched_domains_locked+1304
        c0000000002cbf08 rebuild_sched_domains_locked+216
        c0000000002cfd5c cpuset_handle_hotplug+1148
        c0000000001b3178 sched_cpu_activate+664
        c000000000156fc0 cpuhp_invoke_callback+480
        c000000000157974 cpuhp_thread_fun+244
        c00000000019ddec smpboot_thread_fn+460
        c0000000001932c4 kthread+308
        c00000000000dd58 start_kernel_thread+20
, cpuhp/2, 33]: 1
@[
        80000007642b9b6c 0x80000007642b9b6c
        c000000000206268 partition_sched_domains_locked+1304
        c0000000002cbf08 rebuild_sched_domains_locked+216
        c0000000002cfd5c cpuset_handle_hotplug+1148
        c0000000001b3178 sched_cpu_activate+664
        c000000000156fc0 cpuhp_invoke_callback+480
        c000000000157974 cpuhp_thread_fun+244
        c00000000019ddec smpboot_thread_fn+460
        c0000000001932c4 kthread+308
        c00000000000dd58 start_kernel_thread+20
, cpuhp/3, 38]: 1

+ bpftrace -e 'kfunc:vmlinux:build_sched_domains {@[kstack(raw), comm, tid]=count();}'
Attaching 1 probe...
+ chcpu -d 2-3
CPU 2 disabled
CPU 3 disabled
+ kill 35243

@[
c00800000343346c
, cpuhp/3, 38]: 1
@[
c00800000343346c
, cpuhp/2, 33]: 1


+ bpftrace -e 'kprobe:build_sched_domains {@[kstack(raw), comm, tid]=count();}'
Attaching 1 probe...
+ chcpu -e 2-3
CPU 2 enabled
CPU 3 enabled
+ kill 35249

@[
80000007642befac
c000000000206268
c0000000002cbf08
c0000000002cfd5c
c0000000001b3178
c000000000156fc0
c000000000157974
c00000000019ddec
c0000000001932c4
c00000000000dd58
, cpuhp/3, 38]: 1
@[
80000007524b425c
c000000000206268
c0000000002cbf08
c0000000002cfd5c
c0000000001b3178
c000000000156fc0
c000000000157974
c00000000019ddec
c0000000001932c4
c00000000000dd58
, cpuhp/2, 33]: 1

> 
> 
> Naveen N Rao (11):
>   powerpc/kprobes: Use ftrace to determine if a probe is at function
>     entry
>   powerpc/ftrace: Unify 32-bit and 64-bit ftrace entry code
>   powerpc/module_64: Convert #ifdef to IS_ENABLED()
>   powerpc/ftrace: Remove pointer to struct module from dyn_arch_ftrace
>   kbuild: Add generic hook for architectures to use before the final
>     vmlinux link
>   powerpc64/ftrace: Move ftrace sequence out of line
>   powerpc/ftrace: Add support for DYNAMIC_FTRACE_WITH_CALL_OPS
>   powerpc/ftrace: Add support for DYNAMIC_FTRACE_WITH_DIRECT_CALLS
>   samples/ftrace: Add support for ftrace direct samples on powerpc
>   powerpc64/bpf: Fold bpf_jit_emit_func_call_hlp() into
>     bpf_jit_emit_func_call_rel()
>   powerpc64/bpf: Add support for bpf trampolines
> 
>  arch/Kconfig                                |   3 +
>  arch/powerpc/Kconfig                        |   9 +
>  arch/powerpc/Makefile                       |   8 +
>  arch/powerpc/include/asm/ftrace.h           |  29 +-
>  arch/powerpc/include/asm/module.h           |   5 +
>  arch/powerpc/include/asm/ppc-opcode.h       |  14 +
>  arch/powerpc/kernel/asm-offsets.c           |  11 +
>  arch/powerpc/kernel/kprobes.c               |  18 +-
>  arch/powerpc/kernel/module_64.c             |  67 +-
>  arch/powerpc/kernel/trace/ftrace.c          | 269 +++++++-
>  arch/powerpc/kernel/trace/ftrace_64_pg.c    |  73 +-
>  arch/powerpc/kernel/trace/ftrace_entry.S    | 210 ++++--
>  arch/powerpc/kernel/vmlinux.lds.S           |   3 +-
>  arch/powerpc/net/bpf_jit.h                  |  11 +
>  arch/powerpc/net/bpf_jit_comp.c             | 702 +++++++++++++++++++-
>  arch/powerpc/net/bpf_jit_comp32.c           |   7 +-
>  arch/powerpc/net/bpf_jit_comp64.c           |  68 +-
>  arch/powerpc/tools/Makefile                 |  10 +
>  arch/powerpc/tools/gen-ftrace-pfe-stubs.sh  |  49 ++
>  samples/ftrace/ftrace-direct-modify.c       |  85 ++-
>  samples/ftrace/ftrace-direct-multi-modify.c | 101 ++-
>  samples/ftrace/ftrace-direct-multi.c        |  79 ++-
>  samples/ftrace/ftrace-direct-too.c          |  83 ++-
>  samples/ftrace/ftrace-direct.c              |  69 +-
>  scripts/Makefile.vmlinux                    |   8 +
>  scripts/link-vmlinux.sh                     |  11 +-
>  26 files changed, 1813 insertions(+), 189 deletions(-)
>  create mode 100644 arch/powerpc/tools/Makefile
>  create mode 100755 arch/powerpc/tools/gen-ftrace-pfe-stubs.sh
> 
> 
> base-commit: e2b06d707dd067509cdc9ceba783c06fa6a551c2
> prerequisite-patch-id: a1d50e589288239d5a8b1c1f354cd4737057c9d3
> prerequisite-patch-id: da4142d56880861bd0ad7ad7087c9e2670a2ee54
> prerequisite-patch-id: 609d292e054b2396b603890522a940fa0bdfb6d8
> prerequisite-patch-id: 6f7213fb77b1260defbf43be0e47bff9c80054cc
> prerequisite-patch-id: ad3b71bf071ae4ba1bee5b087e61a2055772a74f
> -- 
> 2.45.2
>