Re: Inconsistent BTF entries for `struct pt_regs *regs` parameter

[Jiri Olsa <olsajiri@gmail.com>]

On Thu, Oct 06, 2022 at 11:54:13AM +0200, Jiri Olsa wrote:
> On Thu, Oct 06, 2022 at 04:34:46PM +0900, Akihiro HARAI wrote:
> > Depending on distribution/kernel/syscall combination, BTF entry for
> > `struct pt_regs *regs` parameter differs.
> > 
> > For example, Amazon Linux 2 with kernel-5.15 package enabled has a FWD
> > entry for `__x64_sys_recvmsg` function:
> > 
> > ```
> > $ uname -a
> > Linux ip-10-1-1-66.ap-northeast-1.compute.internal
> > 5.15.43-20.123.amzn2.x86_64 #1 SMP Fri May 27 00:28:44 UTC 2022 x86_64
> > x86_64 x86_64 GNU/Linux
> > 
> > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> > ...
> > [15439] FWD 'pt_regs' fwd_kind=struct
> > [15440] CONST '(anon)' type_id=15439
> > [15441] PTR '(anon)' type_id=15440
> > [15442] FUNC_PROTO '(anon)' ret_type_id=34 vlen=1
> >         '__unused' type_id=15441
> > ...
> > [15694] FUNC '__x64_sys_recvmsg' type_id=15442 linkage=static
> > ...
> > ```
> > 
> > while Ubuntu 20.04 LTS with newer kernel has a STRUCT entry for the
> > same function:
> > 
> > ```
> > $ uname -a
> > Linux xxx-XPS-13-9300 5.13.0-51-generic #58~20.04.1-Ubuntu SMP Tue Jun
> > 14 11:29:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> > 
> > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> > [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> > ...
> > [226] STRUCT 'pt_regs' size=168 vlen=21
> >         'r15' type_id=1 bits_offset=0
> >         'r14' type_id=1 bits_offset=64
> >         'r13' type_id=1 bits_offset=128
> >         'r12' type_id=1 bits_offset=192
> >         'bp' type_id=1 bits_offset=256
> >         'bx' type_id=1 bits_offset=320
> >         'r11' type_id=1 bits_offset=384
> >         'r10' type_id=1 bits_offset=448
> >         'r9' type_id=1 bits_offset=512
> >         'r8' type_id=1 bits_offset=576
> >         'ax' type_id=1 bits_offset=640
> >         'cx' type_id=1 bits_offset=704
> >         'dx' type_id=1 bits_offset=768
> >         'si' type_id=1 bits_offset=832
> >         'di' type_id=1 bits_offset=896
> >         'orig_ax' type_id=1 bits_offset=960
> >         'ip' type_id=1 bits_offset=1024
> >         'cs' type_id=1 bits_offset=1088
> >         'flags' type_id=1 bits_offset=1152
> >         'sp' type_id=1 bits_offset=1216
> >         'ss' type_id=1 bits_offset=1280
> > ...
> > [5183] CONST '(anon)' type_id=226
> > ...
> > [5189] PTR '(anon)' type_id=5183
> > ...
> > [5321] FUNC_PROTO '(anon)' ret_type_id=42 vlen=1
> >         '__unused' type_id=5189
> > ...
> > [17648] FUNC '__x64_sys_recvmsg' type_id=5321 linkage=static
> > ...
> > ```
> > 
> > Yet another distribution/kernel/syscall combination has multiple `FUNC
> > '__x64_sys_[SYSCALL]'` entries, one for FWD and the other for STRUCT:
> > 
> > ```
> > $ uname -a
> > Linux ip-10-5-0-115.ap-northeast-1.compute.internal
> > 5.10.112-108.499.amzn2.x86_64 #1 SMP Wed Apr 27 23:39:40 UTC 2022
> > x86_64 x86_64 x86_64 GNU/Linux
> > 
> > ```
> > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep
> > __x64_sys_mprotect
> > ...
> > [175] STRUCT 'pt_regs' size=168 vlen=21
> >         'r15' type_id=2 bits_offset=0
> >         'r14' type_id=2 bits_offset=64
> >         'r13' type_id=2 bits_offset=128
> >         'r12' type_id=2 bits_offset=192
> >         'bp' type_id=2 bits_offset=256
> > ...
> > [4215] CONST '(anon)' type_id=175
> > ...
> > [4220] PTR '(anon)' type_id=4215
> > ...
> > [6062] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> >         'regs' type_id=4220
> > ...
> > [11461] FWD 'pt_regs' fwd_kind=struct
> > [11462] CONST '(anon)' type_id=11461
> > [11463] PTR '(anon)' type_id=11462
> > [11464] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> >         '__unused' type_id=11463
> > ...
> > [11698] FUNC '__x64_sys_mprotect' type_id=11464 linkage=static
> > ...
> > [23528] FUNC '__x64_sys_mprotect' type_id=6062 linkage=static
> > ...
> > ```
> > 
> > Trying to read `regs` parameter with FWD entry results in "invalid
> > bpf_context access" error:
> > 
> > ```
> > SEC("fentry/__x64_sys_recvfrom")
> > int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> >   struct event t;
> > 
> >   bpf_get_current_comm(t.comm, TASK_COMM_LEN);
> > 
> >   u64 id = bpf_get_current_pid_tgid();
> >   t.pid = id >> 32;
> > 
> >   // This causes an error on some environments.
> >   t.fd = PT_REGS_PARM1_CORE(regs);
> > 
> >   bpf_printk("comm: %s, pid: %d, fd: %d", t.comm, t.pid, t.fd);
> > 
> >   return 0;
> > ```
> > 
> > ```
> > $ sudo ./output
> > 2022/07/01 03:33:01 loading objects: field FentrySyscall: program
> > fentry_syscall: load program: permission denied:
> >         arg#0 type is not a struct
> >         Unrecognized arg#0 type PTR
> >         ; int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> >         0: (79) r6 = *(u64 *)(r1 +0)
> >         func '__x64_sys_recvfrom' arg0 type FWD is not a struct
> >         invalid bpf_context access off=0 size=8
> >         processed 1 insns (limit 1000000) max_states_per_insn 0
> > total_states 0 peak_states 0 mark_read 0
> > ```
> > 
> > Is this a bug related to toolchain?
> 
> nice, I think it's specific to each object that defines syscall
> 
> if such object has 'struct pt_regs' header with definition included
> it will have full struct pt_regs, if not it will be just fwd ref
> 
> not sure this would break anything else, but change below
> fixes it for me

hi,
did this help? I'll send it as formal patch anyway,
since it fixes the issue for me

jirka

> 
> jirka
> 
> 
> ---
> diff --git a/arch/x86/include/asm/syscall_wrapper.h b/arch/x86/include/asm/syscall_wrapper.h
> index 59358d1bf880..fd2669b1cb2d 100644
> --- a/arch/x86/include/asm/syscall_wrapper.h
> +++ b/arch/x86/include/asm/syscall_wrapper.h
> @@ -6,7 +6,7 @@
>  #ifndef _ASM_X86_SYSCALL_WRAPPER_H
>  #define _ASM_X86_SYSCALL_WRAPPER_H
>  
> -struct pt_regs;
> +#include <asm/ptrace.h>
>  
>  extern long __x64_sys_ni_syscall(const struct pt_regs *regs);
>  extern long __ia32_sys_ni_syscall(const struct pt_regs *regs);