Re: Latest libbpf fails to load programs compiled with old LLVM

[Yonghong Song <yhs@fb.com>]

On 12/4/20 1:34 AM, Toke Høiland-Jørgensen wrote:
> Yonghong Song <yhs@fb.com> writes:
> 
>> On 12/3/20 9:55 AM, Toke Høiland-Jørgensen wrote:
>>> Hi Andrii
>>>
>>> I noticed that recent libbpf versions fail to load BPF files compiled
>>> with old versions of LLVM. E.g., if I compile xdp-tools with LLVM 7 I
>>> get:
>>>
>>> $ sudo ./xdp-loader load testns ../lib/testing/xdp_drop.o -vv
>>> Loading 1 files on interface 'testns'.
>>> libbpf: loading ../lib/testing/xdp_drop.o
>>> libbpf: elf: section(3) prog, size 16, link 0, flags 6, type=1
>>> libbpf: sec 'prog': failed to find program symbol at offset 0
>>> Couldn't open file '../lib/testing/xdp_drop.o': BPF object format invalid
>>>
>>> The 'failed to find program symbol' error seems to have been introduced
>>> with commit c112239272c6 ("libbpf: Parse multi-function sections into
>>> multiple BPF programs").
>>>
>>> Looking at the object file in question, indeed it seems to not have any
>>> function symbols defined:
>>>
>>> $  llvm-objdump --syms ../lib/testing/xdp_drop.o
>>>
>>> ../lib/testing/xdp_drop.o:	file format elf64-bpf
>>>
>>> SYMBOL TABLE:
>>> 0000000000000000 l       .debug_str	0000000000000000
>>> 0000000000000037 l       .debug_str	0000000000000000
>>> 0000000000000042 l       .debug_str	0000000000000000
>>> 0000000000000068 l       .debug_str	0000000000000000
>>> 0000000000000071 l       .debug_str	0000000000000000
>>> 0000000000000076 l       .debug_str	0000000000000000
>>> 000000000000008a l       .debug_str	0000000000000000
>>> 0000000000000097 l       .debug_str	0000000000000000
>>> 00000000000000a3 l       .debug_str	0000000000000000
>>> 00000000000000ac l       .debug_str	0000000000000000
>>> 00000000000000b5 l       .debug_str	0000000000000000
>>> 00000000000000bc l       .debug_str	0000000000000000
>>> 00000000000000c9 l       .debug_str	0000000000000000
>>> 00000000000000d4 l       .debug_str	0000000000000000
>>> 00000000000000dd l       .debug_str	0000000000000000
>>> 00000000000000e1 l       .debug_str	0000000000000000
>>> 00000000000000e5 l       .debug_str	0000000000000000
>>> 00000000000000ea l       .debug_str	0000000000000000
>>> 00000000000000f0 l       .debug_str	0000000000000000
>>> 00000000000000f9 l       .debug_str	0000000000000000
>>> 0000000000000103 l       .debug_str	0000000000000000
>>> 0000000000000113 l       .debug_str	0000000000000000
>>> 0000000000000122 l       .debug_str	0000000000000000
>>> 0000000000000131 l       .debug_str	0000000000000000
>>> 0000000000000000 l    d  prog	0000000000000000 prog
>>> 0000000000000000 l    d  .debug_abbrev	0000000000000000 .debug_abbrev
>>> 0000000000000000 l    d  .debug_info	0000000000000000 .debug_info
>>> 0000000000000000 l    d  .debug_frame	0000000000000000 .debug_frame
>>> 0000000000000000 l    d  .debug_line	0000000000000000 .debug_line
>>> 0000000000000000 g       license	0000000000000000 _license
>>> 0000000000000000 g       prog	0000000000000000 xdp_drop
>>>
>>>
>>> I assume this is because old LLVM versions simply don't emit that symbol
>>> information?

Thanks for the below instruction and xdp_drop.c file. I can reproduce 
the issue now.

I added another function 'xdp_drop1' in the same thing. Below is the
symbol table with llvm7 vs. llvm12.

-bash-4.4$ llvm-readelf -symbols xdp-7.o | grep xdp_drop
     32: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT     3 xdp_drop
     33: 0000000000000010     0 NOTYPE  GLOBAL DEFAULT     3 xdp_drop1

   [ 3] prog              PROGBITS        0000000000000000 000040 000020 
00  AX  0   0  8

-bash-4.4$ llvm-readelf -symbols xdp-12.o | grep xdp_drop
     32: 0000000000000000    16 FUNC    GLOBAL DEFAULT     3 xdp_drop
     33: 0000000000000010    16 FUNC    GLOBAL DEFAULT     3 xdp_drop1
-bash-4.4$

   [ 3] prog              PROGBITS        0000000000000000 000040 000020 
00  AX  0   0  8


Yes, llvm7 does not encode type and size for FUNC's. I guess libbpf can
change to recognize NOTYPE and use the symbol value (representing the 
offset from the start of the section) and section size to
calculate the individual function size. This is more complicated than
elf file providing FUNC type and symbol size directly.

Maybe in this case, libbpf can do some sanity check. If there are more
than one functions in the 'prog' section and they are not marked at FUNC
type, simply recommend newer compiler and bail out saying this feature
not available with old llvm?

>>
>> Could you share xdp_drop.c or other test which I can compile and check
>> to understand the issue?
> 
> It's just an empty program returning XDP_DROP:
> 
> https://github.com/xdp-project/xdp-tools/blob/master/lib/testing/xdp_drop.c
> 
> I basically just did this on Debian buster:
> 
> $ sudo apt install gcc-multilib build-essential libpcap-dev libelf-dev git llc lld clang gcc-multilib pkt-config m4
> $ git clone --recurse-submodules https://github.com/xdp-project/xdp-tools
> $ cd xdp-tools
> $ LLC=llc-7 ./configure
> $ make -k
> $ cd xdp-loader
> $ sudo ip link add dev testns type veth
> $ sudo ./xdp-loader load testns ../lib/testing/xdp_drop.o -vv
> 
> (xdpdump will fail to build with llvm7, but the rest should work)
> 
> -Toke
>