Re: Question: missing vmlinux BTF variable declarations

[Stephen Brennan <stephen@brennan.io>]

Arnaldo Carvalho de Melo <acme@kernel.org> writes:
[...]
>> I think that kallsyms, BTF, and ORC together will be enough to provide a
>> lite debugging experience. Some things will be missing:
>
>> - mapping backtrace addresses to source code lines
>
> So, BTF has provisions for that, and its present in the eBPF programs,
> perf annotate uses it, see tools/perf/util/annotate.c,
> symbol__disassemble_bpf(), it goes like:
>
>         struct bpf_prog_linfo *prog_linfo = NULL;
>
>         info_node = perf_env__find_bpf_prog_info(dso->bpf_prog.env,
>                                                  dso->bpf_prog.id);
>         if (!info_node) {
>                 ret = SYMBOL_ANNOTATE_ERRNO__BPF_MISSING_BTF;
>                 goto out;
>         }
>         info_linear = info_node->info_linear;
>         sub_id = dso->bpf_prog.sub_id;
>
>         info.buffer = (void *)(uintptr_t)(info_linear->info.jited_prog_insns);
>         info.buffer_length = info_linear->info.jited_prog_len;
>
>         if (info_linear->info.nr_line_info)
>                 prog_linfo = bpf_prog_linfo__new(&info_linear->info);
>
>                 addr = pc + ((u64 *)(uintptr_t)(info_linear->info.jited_ksyms))[sub_id];
>                 count = disassemble(pc, &info);
>
>                 if (prog_linfo)
>                         linfo = bpf_prog_linfo__lfind_addr_func(prog_linfo,
>                                                                 addr, sub_id,
>                                                                 nr_skip);
> 		                if (linfo && btf) {
>                         srcline = btf__name_by_offset(btf, linfo->line_off);
>                         nr_skip++;
>                 } else
>                         srcline = NULL;
>
> etc.
>
> Having this for the kernel proper is thus doable, but then we go on
> making BTF info grow.
>
> Perhaps having this as optional, distros or appliances wanting to have a
> kernel with this extra info would add it and then tools would use it if
> available?

I didn't know about the source code mapping support! And I certainly see
the utility of it for BPF programs. However, I'm not sure that a "lite"
kernel debugging experience *needs* source line mapping. I suppose I
should have made it more clear, but I don't think of that list of
"missing" features as a checklist of things we'd want feature parity
for.

The advantage of BTF for debugging would be that it is small, and that
it is part of the kernel image without referencing any other file,
build-id, or kernel version. Ideally, a debugger could load a crash dump
with no additional information, and support a reasonable level of
debugging. I think looking up typed data structure values via global
symbols is part of that level, as well as simple backtraces and other
memory access.

I wouldn't want to try to re-implement DWARF for debuginfo. If you have
the DWARF debuginfo, then your experience should be much better.

>> - intelligent stack frame information from DWARF CFI (e.g.
>>   register/variable values)
>> - probably other things, I'm not a DWARF expert.
[...]
>> > Currently on my local machine, the vmlinux BTF's size is 4.2MB and
>> > adding 1MB would be a big increase. CONFIG_DEBUG_INFO_BTF_ALL is a good
>> > idea. But we might be able to just add global variables without this
>> > new config if we have strong use case.
>  
>> And unfortunately 1MiB is really just a shot in the dark, guessing
>> around 70k variables with no string data.
>
> Maybe we can have a separate BTF file with all this extra info that
> could be fetched from somewhere, keyed by build-id, like is now possible
> with debuginfod and DWARF?

For me, this ranges into the territory of duplicating DWARF. If you lose
the one key advantage of "debuginfoless debugging", then you might as
well use the build-id to lookup DWARF debuginfo as we can today.

This is why I'm trying to propose the means of combining the kallsyms
string data with BTF. Anything that can make the overall size increase
manageable so that all the necessary data can stay in the kernel image.

Thanks,
Stephen