Re: [PATCHv8 bpf-next 28/29] selftests/bpf: Add tracing multi attach benchmark test

[Jiri Olsa <olsajiri@gmail.com>]

On Sun, Jun 07, 2026 at 11:13:32AM -0700, Alexei Starovoitov wrote:
> On Sat Jun 6, 2026 at 5:39 AM PDT, Jiri Olsa wrote:
> > Adding benchmark test that attaches to (almost) all allowed tracing
> > functions and display attach/detach times.
> >
> >   # ./test_progs -t tracing_multi_bench_attach -v
> >   bpf_testmod.ko is already unloaded.
> >   Loading bpf_testmod.ko...
> >   Successfully loaded bpf_testmod.ko.
> >   serial_test_tracing_multi_bench_attach:PASS:btf__load_vmlinux_btf 0 nsec
> >   serial_test_tracing_multi_bench_attach:PASS:tracing_multi_bench__open_and_load 0 nsec
> >   serial_test_tracing_multi_bench_attach:PASS:get_syms 0 nsec
> >   serial_test_tracing_multi_bench_attach:PASS:bpf_program__attach_tracing_multi 0 nsec
> >   serial_test_tracing_multi_bench_attach: found 51186 functions
> >   serial_test_tracing_multi_bench_attach: attached in   1.295s
> >   serial_test_tracing_multi_bench_attach: detached in   0.243s
> 
> ...
> 
> > +	if (!ASSERT_OK(bpf_get_ksyms(&ksyms, true), "get_syms"))
> > +		goto cleanup;
> > +
> > +	/* Get all ftrace 'safe' symbols.. */
> > +	for (i = 0; i < ksyms->filtered_cnt; i++) {
> > +		if (!tsearch(&ksyms->filtered_syms[i], &root, compare)) {
> > +			ASSERT_FAIL("tsearch failed");
> > +			goto cleanup;
> > +		}
> > +	}
> > +
> > +	/* ..and filter them through BTF and btf_type_is_traceable_func. */
> > +	nr = btf__type_cnt(btf);
> > +	for (type_id = 1; type_id < nr; type_id++) {
> > +		const struct btf_type *type;
> > +		const char *str;
> > +
> > +		type = btf__type_by_id(btf, type_id);
> > +		if (!type)
> > +			break;
> > +
> > +		if (BTF_INFO_KIND(type->info) != BTF_KIND_FUNC)
> > +			continue;
> > +
> > +		str = btf__name_by_offset(btf, type->name_off);
> > +		if (!str)
> > +			break;
> > +
> > +		if (!tfind(&str, &root, compare))
> > +			continue;
> > +
> > +		if (!btf_type_is_traceable_func(btf, type))
> > +			continue;
> > +
> > +		err = libbpf_ensure_mem((void **) &ids, &cap, sizeof(*ids), cnt + 1);
> > +		if (err)
> > +			goto cleanup;
> > +
> > +		ids[cnt++] = type_id;
> > +	}
> 
> This filtering wasn't enough.
> I've added removal of duplicates here while applying:
> 
> +       /*
> +        * Collect names that are not unique in kallsyms. The kernel resolves a
> +        * tracing-multi BTF id to an address with kallsyms_lookup_name(), which
> +        * returns the first symbol of that name. For a duplicate name that may
> +        * be a different (non-ftrace-able) instance than the ftrace-able one in
> +        * available_filter_functions, so attaching to it by BTF id fails with
> +        * -ENOENT (e.g. t_start/t_next/t_stop). ksyms->syms is sorted by name,
> +        * so equal names are adjacent.
> +        */
> +       for (i = 1; i < ksyms->sym_cnt; i++) {
> +               if (strcmp(ksyms->syms[i].name, ksyms->syms[i - 1].name))
> +                       continue;
> +               if (!tsearch(&ksyms->syms[i].name, &dups, compare)) {
> +                       ASSERT_FAIL("tsearch failed");
> +                       goto cleanup;
> +               }
> +       }
> 
> 
> +               /* Skip names that are not unique in kallsyms, see above. */
> +               if (tfind(&str, &dups, compare))
> +                       continue;
> 
> 
> As claude explains it:
> ----
> 1. The kernel attaches tracing_multi by BTF id. To get an address it resolves
> the BTF function name via kallsyms_lookup_name(tname) and requires
> ftrace_location(addr) — kernel/bpf/verifier.c:19380:
> addr = kallsyms_lookup_name(tname);
> ...
> if (!addr || !ftrace_location(addr))
>         return -ENOENT;
> 2. t_start/t_next/t_stop each have 5 instances in this kernel. Only one is
> ftrace-able — the copies in kernel/trace/* are built notrace (ftrace's Makefile
> strips -pg), so only the unrelated copy is in available_filter_functions:
> 3. kallsyms_lookup_name() returns the lowest-address instance among equal names
> (exact strcmp, lowest seq). That instance has no fentry → ftrace_location()
> returns 0 → -ENOENT, which aborts the whole all-or-nothing bench attach.
> 
> Why the bench includes them: it intersects BTF FUNC names with
> available_filter_functions names. Since some t_start is ftrace-able, the name
> passes the filter — but the kernel resolves the wrong (non-ftrace-able)
> t_start. The author's kernel apparently had the ftrace-able copy at the lowest
> address, so it passed there.
> 
> This is a pre-existing limitation, not multi-specific: single fentry attach by
> BTF id uses the same kallsyms_lookup_name(tname) path (verifier.c:19120) — you
> can't reliably fentry-attach to any duplicate-named function on this kernel
> either.
> ----

strange I never triggered that.. but makes sense

> 
> Maybe we should adjust bpf_get_ksyms() instead. Not sure.

the only other user is test_kprobe_multi_bench_attach which does
not care about this, so I think at this point keeping this in the
serial_test_tracing_multi_bench_attach is enough for now

thanks,
jirka