Re: Inconsistent BTF entries for `struct pt_regs *regs` parameter

[Jiri Olsa <olsajiri@gmail.com>]

On Thu, Oct 06, 2022 at 04:34:46PM +0900, Akihiro HARAI wrote:
> Depending on distribution/kernel/syscall combination, BTF entry for
> `struct pt_regs *regs` parameter differs.
> 
> For example, Amazon Linux 2 with kernel-5.15 package enabled has a FWD
> entry for `__x64_sys_recvmsg` function:
> 
> ```
> $ uname -a
> Linux ip-10-1-1-66.ap-northeast-1.compute.internal
> 5.15.43-20.123.amzn2.x86_64 #1 SMP Fri May 27 00:28:44 UTC 2022 x86_64
> x86_64 x86_64 GNU/Linux
> 
> $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> ...
> [15439] FWD 'pt_regs' fwd_kind=struct
> [15440] CONST '(anon)' type_id=15439
> [15441] PTR '(anon)' type_id=15440
> [15442] FUNC_PROTO '(anon)' ret_type_id=34 vlen=1
>         '__unused' type_id=15441
> ...
> [15694] FUNC '__x64_sys_recvmsg' type_id=15442 linkage=static
> ...
> ```
> 
> while Ubuntu 20.04 LTS with newer kernel has a STRUCT entry for the
> same function:
> 
> ```
> $ uname -a
> Linux xxx-XPS-13-9300 5.13.0-51-generic #58~20.04.1-Ubuntu SMP Tue Jun
> 14 11:29:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> 
> $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> ...
> [226] STRUCT 'pt_regs' size=168 vlen=21
>         'r15' type_id=1 bits_offset=0
>         'r14' type_id=1 bits_offset=64
>         'r13' type_id=1 bits_offset=128
>         'r12' type_id=1 bits_offset=192
>         'bp' type_id=1 bits_offset=256
>         'bx' type_id=1 bits_offset=320
>         'r11' type_id=1 bits_offset=384
>         'r10' type_id=1 bits_offset=448
>         'r9' type_id=1 bits_offset=512
>         'r8' type_id=1 bits_offset=576
>         'ax' type_id=1 bits_offset=640
>         'cx' type_id=1 bits_offset=704
>         'dx' type_id=1 bits_offset=768
>         'si' type_id=1 bits_offset=832
>         'di' type_id=1 bits_offset=896
>         'orig_ax' type_id=1 bits_offset=960
>         'ip' type_id=1 bits_offset=1024
>         'cs' type_id=1 bits_offset=1088
>         'flags' type_id=1 bits_offset=1152
>         'sp' type_id=1 bits_offset=1216
>         'ss' type_id=1 bits_offset=1280
> ...
> [5183] CONST '(anon)' type_id=226
> ...
> [5189] PTR '(anon)' type_id=5183
> ...
> [5321] FUNC_PROTO '(anon)' ret_type_id=42 vlen=1
>         '__unused' type_id=5189
> ...
> [17648] FUNC '__x64_sys_recvmsg' type_id=5321 linkage=static
> ...
> ```
> 
> Yet another distribution/kernel/syscall combination has multiple `FUNC
> '__x64_sys_[SYSCALL]'` entries, one for FWD and the other for STRUCT:
> 
> ```
> $ uname -a
> Linux ip-10-5-0-115.ap-northeast-1.compute.internal
> 5.10.112-108.499.amzn2.x86_64 #1 SMP Wed Apr 27 23:39:40 UTC 2022
> x86_64 x86_64 x86_64 GNU/Linux
> 
> ```
> $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep
> __x64_sys_mprotect
> ...
> [175] STRUCT 'pt_regs' size=168 vlen=21
>         'r15' type_id=2 bits_offset=0
>         'r14' type_id=2 bits_offset=64
>         'r13' type_id=2 bits_offset=128
>         'r12' type_id=2 bits_offset=192
>         'bp' type_id=2 bits_offset=256
> ...
> [4215] CONST '(anon)' type_id=175
> ...
> [4220] PTR '(anon)' type_id=4215
> ...
> [6062] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
>         'regs' type_id=4220
> ...
> [11461] FWD 'pt_regs' fwd_kind=struct
> [11462] CONST '(anon)' type_id=11461
> [11463] PTR '(anon)' type_id=11462
> [11464] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
>         '__unused' type_id=11463
> ...
> [11698] FUNC '__x64_sys_mprotect' type_id=11464 linkage=static
> ...
> [23528] FUNC '__x64_sys_mprotect' type_id=6062 linkage=static
> ...
> ```
> 
> Trying to read `regs` parameter with FWD entry results in "invalid
> bpf_context access" error:
> 
> ```
> SEC("fentry/__x64_sys_recvfrom")
> int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
>   struct event t;
> 
>   bpf_get_current_comm(t.comm, TASK_COMM_LEN);
> 
>   u64 id = bpf_get_current_pid_tgid();
>   t.pid = id >> 32;
> 
>   // This causes an error on some environments.
>   t.fd = PT_REGS_PARM1_CORE(regs);
> 
>   bpf_printk("comm: %s, pid: %d, fd: %d", t.comm, t.pid, t.fd);
> 
>   return 0;
> ```
> 
> ```
> $ sudo ./output
> 2022/07/01 03:33:01 loading objects: field FentrySyscall: program
> fentry_syscall: load program: permission denied:
>         arg#0 type is not a struct
>         Unrecognized arg#0 type PTR
>         ; int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
>         0: (79) r6 = *(u64 *)(r1 +0)
>         func '__x64_sys_recvfrom' arg0 type FWD is not a struct
>         invalid bpf_context access off=0 size=8
>         processed 1 insns (limit 1000000) max_states_per_insn 0
> total_states 0 peak_states 0 mark_read 0
> ```
> 
> Is this a bug related to toolchain?

nice, I think it's specific to each object that defines syscall

if such object has 'struct pt_regs' header with definition included
it will have full struct pt_regs, if not it will be just fwd ref

not sure this would break anything else, but change below
fixes it for me

jirka


---
diff --git a/arch/x86/include/asm/syscall_wrapper.h b/arch/x86/include/asm/syscall_wrapper.h
index 59358d1bf880..fd2669b1cb2d 100644
--- a/arch/x86/include/asm/syscall_wrapper.h
+++ b/arch/x86/include/asm/syscall_wrapper.h
@@ -6,7 +6,7 @@
 #ifndef _ASM_X86_SYSCALL_WRAPPER_H
 #define _ASM_X86_SYSCALL_WRAPPER_H
 
-struct pt_regs;
+#include <asm/ptrace.h>
 
 extern long __x64_sys_ni_syscall(const struct pt_regs *regs);
 extern long __ia32_sys_ni_syscall(const struct pt_regs *regs);