Re: [PATCH 3/3] bpftool: Allow to link libbpf dynamically

Quentin Monnet
2019-11-27T14:29:53Z

2019-11-27 15:15 UTC+0100 ~ Jiri Olsa <jolsa@redhat.com>
> On Wed, Nov 27, 2019 at 01:38:55PM +0000, Quentin Monnet wrote:
>> 2019-11-27 10:48 UTC+0100 ~ Jiri Olsa <jolsa@kernel.org>
>>> Currently we support only static linking with kernel's libbpf
>>> (tools/lib/bpf). This patch adds LIBBPF_DYNAMIC compile variable
>>> that triggers libbpf detection and bpf dynamic linking:
>>>
>>> $ make -C tools/bpf/bpftool make LIBBPF_DYNAMIC=1
>>>
>>> If libbpf is not installed, build (with LIBBPF_DYNAMIC=1) stops with:
>>>
>>> $ make -C tools/bpf/bpftool LIBBPF_DYNAMIC=1
>>> Auto-detecting system features:
>>> ... libbfd: [ on ]
>>> ... disassembler-four-args: [ on ]
>>> ... zlib: [ on ]
>>> ... libbpf: [ OFF ]
>>>
>>> Makefile:102: *** Error: libbpf-devel is missing, please install it. Stop.
>>>
>>> Adding specific bpftool's libbpf check for libbpf_netlink_open (LIBBPF_0.0.6)
>>> which is the latest we need for bpftool at the moment.
>>>
>>> Adding LIBBPF_DIR compile variable to allow linking with
>>> libbpf installed into specific directory:
>>>
>>> $ make -C tools/lib/bpf/ prefix=/tmp/libbpf/ install_lib install_headers
>>> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/
>>>
>>> It might be needed to clean build tree first because features
>>> framework does not detect the change properly:
>>>
>>> $ make -C tools/build/feature clean
>>> $ make -C tools/bpf/bpftool/ clean
>>>
>>> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
>>> ---
>>> tools/bpf/bpftool/Makefile | 40 ++++++++++++++++++++++++++++++-
>>> tools/build/feature/test-libbpf.c | 9 +++++++
>>> 2 files changed, 48 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
>>> index 39bc6f0f4f0b..2b6ed08cb31e 100644
>>> --- a/tools/bpf/bpftool/Makefile
>>> +++ b/tools/bpf/bpftool/Makefile
>>
>>> @@ -55,7 +64,7 @@ ifneq ($(EXTRA_LDFLAGS),)
>>> LDFLAGS += $(EXTRA_LDFLAGS)
>>> endif
>>>
>>> -LIBS = $(LIBBPF) -lelf -lz
>>> +LIBS = -lelf -lz
>>
>> Hi Jiri,
>>
>> This change seems to be breaking the build with the static library for
>> me. I know you add back $(LIBBPF) later in the Makefile, see at the end
>> of this email...
>>
>>>
>>> INSTALL ?= install
>>> RM ?= rm -f
>>> @@ -64,6 +73,23 @@ FEATURE_USER = .bpftool
>>> FEATURE_TESTS = libbfd disassembler-four-args reallocarray zlib
>>> FEATURE_DISPLAY = libbfd disassembler-four-args zlib
>>>
>>> +ifdef LIBBPF_DYNAMIC
>>> + # Add libbpf check with the flags to ensure bpftool
>>> + # specific version is detected.
>>
>> Nit: We do not check for a specific bpftool version, we check for a
>> recent enough libbpf version?
>
> hi,
> we check for a version that has the latest exported function
> that bpftool needs, which is currently libbpf_netlink_open
>
> please check the '#ifdef BPFTOOL' in feature/test-libbpf.c
>
> it's like that because there's currently no support to check
> for particular library version in the build/features framework
>
> I'll make that comment more clear

Thanks!
>>> + FEATURE_CHECK_CFLAGS-libbpf := -DBPFTOOL
>>> + FEATURE_TESTS += libbpf
>>> + FEATURE_DISPLAY += libbpf
>>> +
>>> + # for linking with debug library run:
>>> + # make LIBBPF_DYNAMIC=1 LIBBPF_DIR=/opt/libbpf
>>> + ifdef LIBBPF_DIR
>>> + LIBBPF_CFLAGS := -I$(LIBBPF_DIR)/include
>>> + LIBBPF_LDFLAGS := -L$(LIBBPF_DIR)/$(libdir_relative)
>>> + FEATURE_CHECK_CFLAGS-libbpf := $(LIBBPF_CFLAGS)
>>> + FEATURE_CHECK_LDFLAGS-libbpf := $(LIBBPF_LDFLAGS)
>>> + endif
>>> +endif
>>> +
>>> check_feat := 1
>>> NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall
>>> ifdef MAKECMDGOALS
>>> @@ -88,6 +114,18 @@ ifeq ($(feature-reallocarray), 0)
>>> CFLAGS += -DCOMPAT_NEED_REALLOCARRAY
>>> endif
>>>
>>> +ifdef LIBBPF_DYNAMIC
>>> + ifeq ($(feature-libbpf), 1)
>>> + LIBS += -lbpf
>>> + CFLAGS += $(LIBBPF_CFLAGS)
>>> + LDFLAGS += $(LIBBPF_LDFLAGS)
>>> + else
>>> + dummy := $(error Error: No libbpf devel library found, please install libbpf-devel)
>>
>> libbpf-devel sounds like a RH/Fedora package name, but other
>> distributions might have different names (Debian/Ubuntu would go by
>> libbpf-dev I suppose, although I don't believe such package exists at
>> the moment). Maybe use a more generic message?
>
> sure, actually in perf we use both package names like:
>
> Error: No libbpf devel library found, please install libbpf-devel or libbpf-dev.
>
> or we can go with generic message:
>
> Error: No libbpf devel library found, please install.

Either is fine with me, thanks!

>>> + endif
>>> +else
>>> + LIBS += $(LIBBPF)
>>
>> ... I believe the order of the libraries is relevant, and it seems the
>> static libbpf should be passed before the dynamic libs. Here I could fix
>> the build with the static library on my setup by prepending the library
>> path instead, like this:
>>
>> LIBS := $(LIBBPF) $(LIBS)
>
> could you please paste the build error? I don't see any on Fedora,

Sure, here it is. On top of your branch (on Ubuntu 18.04):

----------------

$ make -C tools/bpf/bpftool/
make: Entering directory '/root/linux/bpf-next/tools/bpf/bpftool'

Auto-detecting system features:
... libbfd: [ on ]
... disassembler-four-args: [ on ]
... zlib: [ on ]

CC map_perf_ring.o
CC xlated_dumper.o
CC btf.o
CC tracelog.o
CC perf.o
CC cfg.o
CC btf_dumper.o
CC net.o
CC netlink_dumper.o
CC common.o
CC cgroup.o
CC main.o
CC json_writer.o
CC prog.o
CC map.o
CC feature.o
CC jit_disasm.o
CC disasm.o
make[1]: Entering directory '/root/linux/bpf-next/tools/lib/bpf'

Auto-detecting system features:
... libelf: [ on ]
... bpf: [ on ]

Parsed description of 117 helper function(s)
MKDIR staticobjs/
CC staticobjs/libbpf.o
CC staticobjs/bpf.o
CC staticobjs/nlattr.o
CC staticobjs/btf.o
CC staticobjs/libbpf_errno.o
CC staticobjs/str_error.o
CC staticobjs/netlink.o
CC staticobjs/bpf_prog_linfo.o
CC staticobjs/libbpf_probes.o
CC staticobjs/xsk.o
CC staticobjs/hashmap.o
CC staticobjs/btf_dump.o
LD staticobjs/libbpf-in.o
LINK libbpf.a
make[1]: Leaving directory '/root/linux/bpf-next/tools/lib/bpf'
LINK bpftool
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__init_prog_names':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:466: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:473: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__elf_finish':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:570: undefined reference to `elf_end'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__elf_init':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:600: undefined reference to `elf_memory'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:613: undefined reference to `elf_begin'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:623: undefined reference to `gelf_getehdr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object_search_section_size':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:714: undefined reference to `gelf_getshdr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:720: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:730: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:708: undefined reference to `elf_nextscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__variable_offset':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:784: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:790: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__init_user_maps':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:937: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:939: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:957: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:981: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:990: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__init_user_btf_maps':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1359: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1361: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `section_have_execinstr':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1428: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1432: undefined reference to `gelf_getshdr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__elf_collect':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1608: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1608: undefined reference to `elf_rawdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1619: undefined reference to `gelf_getshdr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1625: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1632: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1613: undefined reference to `elf_nextscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_program__collect_reloc':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1928: undefined reference to `gelf_getrel'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1932: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1941: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `__bpf_object__open':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:3934: undefined reference to `elf_version'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `btf__parse_elf':
/root/linux/bpf-next/tools/lib/bpf/btf.c:413: undefined reference to `elf_version'
/root/linux/bpf-next/tools/lib/bpf/btf.c:427: undefined reference to `elf_begin'
/root/linux/bpf-next/tools/lib/bpf/btf.c:432: undefined reference to `gelf_getehdr'
/root/linux/bpf-next/tools/lib/bpf/btf.c:440: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/btf.c:440: undefined reference to `elf_rawdata'
/root/linux/bpf-next/tools/lib/bpf/btf.c:450: undefined reference to `gelf_getshdr'
/root/linux/bpf-next/tools/lib/bpf/btf.c:455: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/btf.c:462: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/btf.c:470: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/btf.c:445: undefined reference to `elf_nextscn'
/root/linux/bpf-next/tools/lib/bpf/btf.c:500: undefined reference to `elf_end'
collect2: error: ld returned 1 exit status
Makefile:158: recipe for target 'bpftool' failed
make: *** [bpftool] Error 1
make: Leaving directory '/root/linux/bpf-next/tools/bpf/bpftool'

----------------

Adding the V=1 flag shows the compilation is attempted with the
following command:

gcc -O2 -W -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wbad-function-cast -Wdeclaration-after-statement -Wformat-security -Wformat-y2k -Winit-self -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wno-system-headers -Wold-style-definition -Wpacked -Wredundant-decls -Wstrict-prototypes -Wswitch-default -Wundef -Wwrite-strings -Wformat -Wstrict-aliasing=3 -Wshadow -DPACKAGE='"bpftool"' -D__EXPORTED_HEADERS__ -I/root/linux/bpf-next/kernel/bpf/ -I/root/linux/bpf-next/tools/include -I/root/linux/bpf-next/tools/include/uapi -I/root/linux/bpf-next/tools/lib/bpf -I/root/linux/bpf-next/tools/perf -DBPFTOOL_VERSION='"5.4.0"' -DDISASM_FOUR_ARGS_SIGNATURE -DHAVE_LIBBFD_SUPPORT -o bpftool map_perf_ring.o perf.o net.o cgroup.o tracelog.o btf_dumper.o xlated_dumper.o json_writer.o prog.o map.o common.o btf.o cfg.o main.o netlink_dumper.o feature.o jit_disasm.o disasm.o -lelf -lz /root/linux/bpf-next/tools/lib/bpf/libbpf.a -lbfd -ldl -lopcodes

If I move -lelf and -lz after <path>/libbpf.a, then it builds fine.

Hope this helps,
Quentin