Re: bpf: Question about odd BPF verifier behaviour

[Eduard Zingerman <eddyz87@gmail.com>]

On Mon, 2023-02-27 at 11:24 -0800, Andrii Nakryiko wrote:
> On Mon, Feb 27, 2023 at 10:05 AM KP Singh <kpsingh@kernel.org> wrote:
> > 
> > On Mon, Feb 27, 2023 at 6:32 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > > 
> > > On Mon, Feb 27, 2023 at 6:17 AM Eduard Zingerman <eddyz87@gmail.com> wrote:
> > > > 
> > > > On Sun, 2023-02-26 at 03:03 +0200, Eduard Zingerman wrote:
> > > > > On Sat, 2023-02-25 at 20:50 +0000, Matt Bobrowski wrote:
> > > > > > Sorry Eduard, I replied late last night although the email bounced due
> > > > > > to exceeding the mail char limit. Let's try attaching a compressed
> > > > > > variant of the requested files, which includes the compiled kernel's
> > > > > > BTF and the kernel's config.
> > > > > 
> > > > > Hi Matt,
> > > > > 
> > > > > I tried using your config but still can't reproduce the issue.
> > > > > Will try to do it using debian 12 chroot tomorrow or on Monday.
> > > > 
> > > > Hi Matt,
> > > > 
> > > > Short update:
> > > > I've reproduced the issue with multiple STRUCT 'linux_binprm' BTF IDs
> > > > in Debian testing chroot, thank you for providing all details.
> > > > Attaching the instructions in the end of the email.
> > > > Need some time to analyze pahole behavior.
> > > > 
> > > 
> > > Try using [0] to pinpoint what actually is different between any two
> > > linux_binprm definitions. I've hacked up this "tool" last time I had
> > > to pinpoint where two BTF types diverge, maybe it will save you a bit
> > > of time as well. I'd like to put this functionality into btfdump
> > > ([1]), but I didn't get to it yet, unfortunately.
> > > 
> > 
> > It's not just linux_binprm but a bunch of other structs (quite a lot
> > of them that seem to diverge)
> 
> yes, task_struct is usually a thing that suffers from such
> duplications, as it forms a huge graph of types. Which is where that
> btfdiff "tool" comes handly, it recursively compares side-by-side two
> types that are supposed to be equal (but are not), by ignoring BTF
> type IDs and trying to pin point actual differences (like STRUCT vs
> FWD, or extra field, or whatever). It just needs two starting type
> IDs.

Thank you for links to these tools!

> 
> > 
> > [...]
> > WARN: multiple IDs found for 'sock_common': 4400, 53212 - using 4400
> > WARN: multiple IDs found for 'request_sock': 4458, 53257 - using 4458
> > WARN: multiple IDs found for 'task_struct': 265, 55830 - using 265
> > WARN: multiple IDs found for 'file': 474, 55854 - using 474
> > WARN: multiple IDs found for 'vm_area_struct': 480, 55857 - using 480
> > WARN: multiple IDs found for 'seq_file': 670, 55891 - using 670
> 
> 
> [...]
> 
> > 
> > I was able to "fix" this with using clang 16.x and the following hacky pahole:
> > 
> > 030e3b4 - core: Add DW_TAG_unspecified_type to tag__is_tag_type() set
> > (HEAD) (2023-02-26 Arnaldo Carvalho de Melo)
> > f20515b - dwarves: support DW_TAG_atomic_type (2023-02-26 David Lamparter)
> > de24234 - pahole: Prep 1.24 (tag: v1.24) (2022-08-22 Arnaldo Carvalho de Melo)
> > d6c9528 - dwarf_loader: Encode char type as signed (2022-08-10 Yonghong Song)
> > 
> > and the the following patch on top:
> > 
> 
> I'd start with understanding what BTF and DWARF differences are
> causing the issue before trying to come up with the fix. For that we
> don't even need config or repro steps, it should be enough to share
> vmlinux with BTF and DWARF, and start from there.
> 

Yes, I suspect that there is some kind of unanticipated
anomaly for some DWARF encoding for some kind of objects,
just need to find the root for the diverging type hierarchies. 

> But I'm sure Eduard is on top of this already (especially that he can
> repro the issue now).

I'm working on it, nothing to report yet, but I'm working on it.