Re: [PATCH bpf-next 1/4] bpf: Allow get_func_[arg|arg_cnt] helpers in raw tracepoint programs

[Leon Hwang <leon.hwang@linux.dev>]

On 2025/5/1 00:53, Alexei Starovoitov wrote:
> On Wed, Apr 30, 2025 at 8:55 AM Leon Hwang <leon.hwang@linux.dev> wrote:
>>
>>
>>
>> On 2025/4/30 20:43, Kafai Wan wrote:
>>> On Wed, Apr 30, 2025 at 10:46 AM Alexei Starovoitov
>>> <alexei.starovoitov@gmail.com> wrote:
>>>>
>>>> On Sat, Apr 26, 2025 at 9:00 AM KaFai Wan <mannkafai@gmail.com> wrote:
>>>>>
>>

[...]

>>
>>
>> bpf_get_func_arg() will be very helpful for bpfsnoop[1] when tracing tp_btf.
>>
>> In bpfsnoop, it can generate a small snippet of bpf instructions to use
>> bpf_get_func_arg() for retrieving and filtering arguments. For example,
>> with the netif_receive_skb tracepoint, bpfsnoop can use
>> bpf_get_func_arg() to filter the skb argument using pcap-filter(7)[2] or
>> a custom attribute-based filter. This will allow bpfsnoop to trace
>> multiple tracepoints using a single bpf program code.
> 
> I doubt you thought it through end to end.
> When tracepoint prog attaches we have this check:
>         /*
>          * check that program doesn't access arguments beyond what's
>          * available in this tracepoint
>          */
>         if (prog->aux->max_ctx_offset > btp->num_args * sizeof(u64))
>                 return -EINVAL;
> 
> So you cannot have a single bpf prog attached to many tracepoints
> to read many arguments as-is.
> You can hack around that limit with probe_read,
> but the values won't be trusted and you won't be able to pass
> such untrusted pointers into skb and other helpers/kfuncs.

I understand that a single bpf program cannot be attached to multiple
tracepoints using tp_btf. However, the same bpf code can be reused to
create multiple bpf programs, each attached to a different tracepoint.

For example:

SEC("fentry")
int BPF_PROG(fentry_fn)
{
	/* ... */
	return BPF_OK;
}

The above fentry code can be compiled into multiple bpf programs to
trace different kernel functions. Each program can then use the
bpf_get_func_arg() helper to access the arguments of the traced function.

With this patch, tp_btf will gain similar flexibility. For example:

SEC("tp_btf")
int BPF_PROG(tp_btf_fn)
{
	/* ... */
	return BPF_OK;
}

Here, bpf_get_func_arg() can be used to access tracepoint arguments.

Currently, due to the lack of bpf_get_func_arg() support in tp_btf,
bpfsnoop[1] uses bpf_probe_read_kernel() to read tracepoint arguments.
This is also used when filtering specific argument attributes.

For instance, to filter the skb argument of the netif_receive_skb
tracepoint by 'skb->dev->ifindex == 2', the translated bpf instructions
with bpf_probe_read_kernel() would look like this:

bool filter_arg(__u64 * args):
; filter_arg(__u64 *args)
 209: (79) r1 = *(u64 *)(r1 +0) /* all tracepoint's argument has been
read into args using bpf_probe_read_kernel() */
 210: (bf) r3 = r1
 211: (07) r3 += 16
 212: (b7) r2 = 8
 213: (bf) r1 = r10
 214: (07) r1 += -8
 215: (85) call bpf_probe_read_kernel#-125280
 216: (79) r3 = *(u64 *)(r10 -8)
 217: (15) if r3 == 0x0 goto pc+10
 218: (07) r3 += 224
 219: (b7) r2 = 8
 220: (bf) r1 = r10
 221: (07) r1 += -8
 222: (85) call bpf_probe_read_kernel#-125280
 223: (79) r3 = *(u64 *)(r10 -8)
 224: (67) r3 <<= 32
 225: (77) r3 >>= 32
 226: (b7) r0 = 1
 227: (15) if r3 == 0x2 goto pc+1
 228: (af) r0 ^= r0
 229: (95) exit

If bpf_get_func_arg() is supported in tp_btf, the bpf program will
instead look like:

static __noinline bool
filter_skb(void *ctx)
{
    struct sk_buff *skb;

    (void) bpf_get_func_arg(ctx, 0, (__u64 *) &skb);
    return skb->dev->ifindex == 2;
}

This will simplify the generated code and eliminate the need for
bpf_probe_read_kernel() calls. However, in my tests (on kernel
6.8.0-35-generic, Ubuntu 24.04 LTS), the pointer returned by
bpf_get_func_arg() is marked as a scalar rather than a trusted pointer:

	0: R1=ctx() R10=fp0
	; if (!filter_skb(ctx))
	0: (85) call pc+3
	caller:
	 R10=fp0
	callee:
	 frame1: R1=ctx() R10=fp0
	4: frame1: R1=ctx() R10=fp0
	; filter_skb(void *ctx)
	4: (bf) r3 = r10                      ; frame1: R3_w=fp0 R10=fp0
	;
	5: (07) r3 += -8                      ; frame1: R3_w=fp-8
	; (void) bpf_get_func_arg(ctx, 0, (__u64 *) &skb);
	6: (b7) r2 = 0                        ; frame1: R2_w=0
	7: (85) call bpf_get_func_arg#183     ; frame1: R0_w=scalar()
	; return skb->dev->ifindex == 2;
	8: (79) r1 = *(u64 *)(r10 -8)         ; frame1: R1_w=scalar() R10=fp0
fp-8=mmmmmmmm
	; return skb->dev->ifindex == 2;
	9: (79) r1 = *(u64 *)(r1 +16)
	R1 invalid mem access 'scalar'
	processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0
peak_states 0 mark_read 0

If the returned skb is a trusted pointer, the verifier will accept
something like:

static __noinline bool
filter_skb(struct sk_buff *skb)
{
    return skb->dev->ifindex == 2;
}

Which will compile into much simpler and more efficient instructions:

bool filter_skb(struct sk_buff * skb):
; return skb->dev->ifindex == 2;
  92: (79) r1 = *(u64 *)(r1 +16)
; return skb->dev->ifindex == 2;
  93: (61) r1 = *(u32 *)(r1 +224)
  94: (b7) r0 = 1
; return skb->dev->ifindex == 2;
  95: (15) if r1 == 0x2 goto pc+1
  96: (b7) r0 = 0
; return skb->dev->ifindex == 2;
  97: (95) exit

In conclusion:

1. It will be better if the pointer returned by bpf_get_func_arg() is
trusted, only when the argument index is a known constant.
2. Adding bpf_get_func_arg() support to tp_btf will significantly
simplify and improve tools like bpfsnoop.

[1] https://github.com/bpfsnoop/bpfsnoop

Thanks,
Leon