* Re: [PATCH bpf-next] selftests/bpf: Cross-compile bpftool
2023-02-10 8:43 [PATCH bpf-next] selftests/bpf: Cross-compile bpftool Björn Töpel
@ 2023-02-13 14:30 ` Björn Töpel
2023-02-14 9:16 ` Zachary Leaf
2023-02-13 20:53 ` Quentin Monnet
2023-02-14 8:57 ` Zachary Leaf
2 siblings, 1 reply; 8+ messages in thread
From: Björn Töpel @ 2023-02-13 14:30 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, bpf, netdev
Cc: Björn Töpel, Jean-Philippe Brucker, linux-kselftest,
Anders Roxell, linux-riscv
Björn Töpel <bjorn@kernel.org> writes:
> From: Björn Töpel <bjorn@rivosinc.com>
>
> When the BPF selftests are cross-compiled, only the a host version of
> bpftool is built. This version of bpftool is used to generate various
> intermediates, e.g., skeletons.
>
> The test runners are also using bpftool. The Makefile will symlink
> bpftool from the selftest/bpf root, where the test runners will look
> for the tool:
>
> | ...
> | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \
> | $(OUTPUT)/$(if $2,$2/)bpftool
>
> There are two issues for cross-compilation builds:
>
> 1. There is no native (cross-compilation target) build of bpftool
> 2. The bootstrap variant of bpftool is never cross-compiled (by
> design)
>
> Make sure that a native/cross-compiled version of bpftool is built,
> and if CROSS_COMPILE is set, symlink to the native/non-bootstrap
> version.
...and the grand master plan is to add BPF CI support for riscv64, where
this patch a prerequisite to [1]. I would suspect that other platforms
might benefit from cross-compilation builds as well.
[1] https://github.com/kernel-patches/vmtest/pull/194
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH bpf-next] selftests/bpf: Cross-compile bpftool
2023-02-13 14:30 ` Björn Töpel
@ 2023-02-14 9:16 ` Zachary Leaf
2023-02-14 9:44 ` Björn Töpel
0 siblings, 1 reply; 8+ messages in thread
From: Zachary Leaf @ 2023-02-14 9:16 UTC (permalink / raw)
To: Björn Töpel, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, bpf, netdev
Cc: Björn Töpel, Jean-Philippe Brucker, linux-kselftest,
Anders Roxell, linux-riscv, Quentin Monnet, linux-morello
On 13/02/2023 14:30, Björn Töpel wrote:
> Björn Töpel <bjorn@kernel.org> writes:
>
>> From: Björn Töpel <bjorn@rivosinc.com>
>>
>> When the BPF selftests are cross-compiled, only the a host version of
>> bpftool is built. This version of bpftool is used to generate various
>> intermediates, e.g., skeletons.
>>
>> The test runners are also using bpftool. The Makefile will symlink
>> bpftool from the selftest/bpf root, where the test runners will look
>> for the tool:
>>
>> | ...
>> | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \
>> | $(OUTPUT)/$(if $2,$2/)bpftool
>>
>> There are two issues for cross-compilation builds:
>>
>> 1. There is no native (cross-compilation target) build of bpftool
>> 2. The bootstrap variant of bpftool is never cross-compiled (by
>> design)
>>
>> Make sure that a native/cross-compiled version of bpftool is built,
>> and if CROSS_COMPILE is set, symlink to the native/non-bootstrap
>> version.
>
> ...and the grand master plan is to add BPF CI support for riscv64, where
> this patch a prerequisite to [1]. I would suspect that other platforms
> might benefit from cross-compilation builds as well.
Similar use case. There also seems to be a lot of issues building these
tests out of tree.
I have some potential fixes up to 6.1 but linux-next seems to have
introduced a few more issues on top.
>
> [1] https://github.com/kernel-patches/vmtest/pull/194
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH bpf-next] selftests/bpf: Cross-compile bpftool
2023-02-14 9:16 ` Zachary Leaf
@ 2023-02-14 9:44 ` Björn Töpel
0 siblings, 0 replies; 8+ messages in thread
From: Björn Töpel @ 2023-02-14 9:44 UTC (permalink / raw)
To: Zachary Leaf, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, bpf, netdev
Cc: Björn Töpel, Jean-Philippe Brucker, linux-kselftest,
Anders Roxell, linux-riscv, Quentin Monnet, linux-morello
Zachary Leaf <zachary.leaf@arm.com> writes:
> On 13/02/2023 14:30, Björn Töpel wrote:
>> Björn Töpel <bjorn@kernel.org> writes:
>>
>>> From: Björn Töpel <bjorn@rivosinc.com>
>>>
>>> When the BPF selftests are cross-compiled, only the a host version of
>>> bpftool is built. This version of bpftool is used to generate various
>>> intermediates, e.g., skeletons.
>>>
>>> The test runners are also using bpftool. The Makefile will symlink
>>> bpftool from the selftest/bpf root, where the test runners will look
>>> for the tool:
>>>
>>> | ...
>>> | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \
>>> | $(OUTPUT)/$(if $2,$2/)bpftool
>>>
>>> There are two issues for cross-compilation builds:
>>>
>>> 1. There is no native (cross-compilation target) build of bpftool
>>> 2. The bootstrap variant of bpftool is never cross-compiled (by
>>> design)
>>>
>>> Make sure that a native/cross-compiled version of bpftool is built,
>>> and if CROSS_COMPILE is set, symlink to the native/non-bootstrap
>>> version.
>>
>> ...and the grand master plan is to add BPF CI support for riscv64, where
>> this patch a prerequisite to [1]. I would suspect that other platforms
>> might benefit from cross-compilation builds as well.
>
> Similar use case. There also seems to be a lot of issues building these
> tests out of tree.
>
> I have some potential fixes up to 6.1 but linux-next seems to have
> introduced a few more issues on top.
Ah, yes. FWIW, the BPF CI builds the selftests *in-tree*, so with this
patch (and my PRs) the BPF CI is capable of cross-compiling.
Björn
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH bpf-next] selftests/bpf: Cross-compile bpftool
2023-02-10 8:43 [PATCH bpf-next] selftests/bpf: Cross-compile bpftool Björn Töpel
2023-02-13 14:30 ` Björn Töpel
@ 2023-02-13 20:53 ` Quentin Monnet
2023-02-14 9:46 ` Jean-Philippe Brucker
2023-02-14 8:57 ` Zachary Leaf
2 siblings, 1 reply; 8+ messages in thread
From: Quentin Monnet @ 2023-02-13 20:53 UTC (permalink / raw)
To: Björn Töpel, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, bpf, netdev, Jean-Philippe Brucker
Cc: Björn Töpel, linux-kselftest
2023-02-10 09:43 UTC+0100 ~ Björn Töpel <bjorn@kernel.org>
> From: Björn Töpel <bjorn@rivosinc.com>
>
> When the BPF selftests are cross-compiled, only the a host version of
> bpftool is built. This version of bpftool is used to generate various
> intermediates, e.g., skeletons.
>
> The test runners are also using bpftool. The Makefile will symlink
> bpftool from the selftest/bpf root, where the test runners will look
> for the tool:
>
> | ...
> | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \
> | $(OUTPUT)/$(if $2,$2/)bpftool
>
> There are two issues for cross-compilation builds:
>
> 1. There is no native (cross-compilation target) build of bpftool
> 2. The bootstrap variant of bpftool is never cross-compiled (by
> design)
>
> Make sure that a native/cross-compiled version of bpftool is built,
> and if CROSS_COMPILE is set, symlink to the native/non-bootstrap
> version.
>
> Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
> ---
> tools/testing/selftests/bpf/Makefile | 28 +++++++++++++++++++++++++---
> 1 file changed, 25 insertions(+), 3 deletions(-)
>
> diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
> index b2eb3201b85a..b706750f71e2 100644
> --- a/tools/testing/selftests/bpf/Makefile
> +++ b/tools/testing/selftests/bpf/Makefile
> @@ -157,8 +157,9 @@ $(notdir $(TEST_GEN_PROGS) \
> $(TEST_CUSTOM_PROGS)): %: $(OUTPUT)/% ;
>
> # sort removes libbpf duplicates when not cross-building
> -MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
> - $(HOST_BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/resolve_btfids \
> +MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
> + $(BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/bpftool \
> + $(HOST_BUILD_DIR)/resolve_btfids \
> $(RUNQSLOWER_OUTPUT) $(INCLUDE_DIR))
> $(MAKE_DIRS):
> $(call msg,MKDIR,,$@)
> @@ -208,6 +209,14 @@ $(OUTPUT)/bpf_testmod.ko: $(VMLINUX_BTF) $(wildcard bpf_testmod/Makefile bpf_tes
> $(Q)cp bpf_testmod/bpf_testmod.ko $@
>
> DEFAULT_BPFTOOL := $(HOST_SCRATCH_DIR)/sbin/bpftool
> +ifneq ($(CROSS_COMPILE),)
> +CROSS_BPFTOOL := $(SCRATCH_DIR)/sbin/bpftool
> +TRUNNER_BPFTOOL := $(CROSS_BPFTOOL)
> +USE_BOOTSTRAP := ""
> +else
> +TRUNNER_BPFTOOL := $(DEFAULT_BPFTOOL)
> +USE_BOOTSTRAP := "bootstrap"
> +endif
>
> $(OUTPUT)/runqslower: $(BPFOBJ) | $(DEFAULT_BPFTOOL) $(RUNQSLOWER_OUTPUT)
> $(Q)$(MAKE) $(submake_extras) -C $(TOOLSDIR)/bpf/runqslower \
> @@ -255,6 +264,18 @@ $(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
> LIBBPF_DESTDIR=$(HOST_SCRATCH_DIR)/ \
> prefix= DESTDIR=$(HOST_SCRATCH_DIR)/ install-bin
>
> +ifneq ($(CROSS_COMPILE),)
> +$(CROSS_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
> + $(BPFOBJ) | $(BUILD_DIR)/bpftool
> + $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
> + ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) \
> + EXTRA_CFLAGS='-g -O0' \
> + OUTPUT=$(BUILD_DIR)/bpftool/ \
> + LIBBPF_OUTPUT=$(BUILD_DIR)/libbpf/ \
> + LIBBPF_DESTDIR=$(SCRATCH_DIR)/ \
> + prefix= DESTDIR=$(SCRATCH_DIR)/ install-bin
> +endif
> +
> all: docs
>
> docs:
> @@ -518,11 +539,12 @@ endif
> $(OUTPUT)/$(TRUNNER_BINARY): $(TRUNNER_TEST_OBJS) \
> $(TRUNNER_EXTRA_OBJS) $$(BPFOBJ) \
> $(RESOLVE_BTFIDS) \
> + $(TRUNNER_BPFTOOL) \
> | $(TRUNNER_BINARY)-extras
> $$(call msg,BINARY,,$$@)
> $(Q)$$(CC) $$(CFLAGS) $$(filter %.a %.o,$$^) $$(LDLIBS) -o $$@
> $(Q)$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.bpf.o $$@
> - $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \
> + $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/$(USE_BOOTSTRAP)/bpftool \
Nit: You'll have a double slash in this path when USE_BOOSTRAP is empty
(.../tools/build/bpftool//bpftool), but it probably doesn't matter much.
> $(OUTPUT)/$(if $2,$2/)bpftool
>
> endef
>
> base-commit: 06744f24696e1e7598412c3df61a538b57ebec22
The changes look good to me, thanks!
Acked-by: Quentin Monnet <quentin@isovalent.com>
Jean-Philippe, I know you do some cross-compiling with bpftool, how does
this look from your side?
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH bpf-next] selftests/bpf: Cross-compile bpftool
2023-02-13 20:53 ` Quentin Monnet
@ 2023-02-14 9:46 ` Jean-Philippe Brucker
0 siblings, 0 replies; 8+ messages in thread
From: Jean-Philippe Brucker @ 2023-02-14 9:46 UTC (permalink / raw)
To: Quentin Monnet
Cc: Björn Töpel, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, bpf, netdev, Björn Töpel,
linux-kselftest
Hi Quentin,
On Mon, Feb 13, 2023 at 08:53:31PM +0000, Quentin Monnet wrote:
> Jean-Philippe, I know you do some cross-compiling with bpftool, how does
> this look from your side?
I don't have enough time for BPF at the moment unfortunately. Zachary is
looking at cross-compiling the selftests for arm64
Thanks,
Jean
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH bpf-next] selftests/bpf: Cross-compile bpftool
2023-02-10 8:43 [PATCH bpf-next] selftests/bpf: Cross-compile bpftool Björn Töpel
2023-02-13 14:30 ` Björn Töpel
2023-02-13 20:53 ` Quentin Monnet
@ 2023-02-14 8:57 ` Zachary Leaf
2023-02-14 9:41 ` Björn Töpel
2 siblings, 1 reply; 8+ messages in thread
From: Zachary Leaf @ 2023-02-14 8:57 UTC (permalink / raw)
To: Björn Töpel, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, bpf, netdev
Cc: Björn Töpel, Jean-Philippe Brucker, linux-kselftest,
Anders Roxell, linux-morello, Quentin Monnet
Hi Bjorn,
Thanks for the patch, I've tested it and it works for me.
I have a minor suggestion but otherwise happy to see this getting fixed.
On 10/02/2023 08:43, Björn Töpel wrote:
> From: Björn Töpel <bjorn@rivosinc.com>
>
> When the BPF selftests are cross-compiled, only the a host version of
> bpftool is built. This version of bpftool is used to generate various
> intermediates, e.g., skeletons.
>
> The test runners are also using bpftool. The Makefile will symlink
> bpftool from the selftest/bpf root, where the test runners will look
> for the tool:
>
> | ...
> | $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \
> | $(OUTPUT)/$(if $2,$2/)bpftool
>
> There are two issues for cross-compilation builds:
>
> 1. There is no native (cross-compilation target) build of bpftool
> 2. The bootstrap variant of bpftool is never cross-compiled (by
> design)
>
> Make sure that a native/cross-compiled version of bpftool is built,
> and if CROSS_COMPILE is set, symlink to the native/non-bootstrap
> version.
>
> Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
> ---
> tools/testing/selftests/bpf/Makefile | 28 +++++++++++++++++++++++++---
> 1 file changed, 25 insertions(+), 3 deletions(-)
>
> diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
> index b2eb3201b85a..b706750f71e2 100644
> --- a/tools/testing/selftests/bpf/Makefile
> +++ b/tools/testing/selftests/bpf/Makefile
> @@ -157,8 +157,9 @@ $(notdir $(TEST_GEN_PROGS) \
> $(TEST_CUSTOM_PROGS)): %: $(OUTPUT)/% ;
>
> # sort removes libbpf duplicates when not cross-building
> -MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
> - $(HOST_BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/resolve_btfids \
> +MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
> + $(BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/bpftool \
> + $(HOST_BUILD_DIR)/resolve_btfids \
> $(RUNQSLOWER_OUTPUT) $(INCLUDE_DIR))
> $(MAKE_DIRS):
> $(call msg,MKDIR,,$@)
> @@ -208,6 +209,14 @@ $(OUTPUT)/bpf_testmod.ko: $(VMLINUX_BTF) $(wildcard bpf_testmod/Makefile bpf_tes
> $(Q)cp bpf_testmod/bpf_testmod.ko $@
>
> DEFAULT_BPFTOOL := $(HOST_SCRATCH_DIR)/sbin/bpftool
> +ifneq ($(CROSS_COMPILE),)
> +CROSS_BPFTOOL := $(SCRATCH_DIR)/sbin/bpftool
> +TRUNNER_BPFTOOL := $(CROSS_BPFTOOL)
> +USE_BOOTSTRAP := ""
> +else
> +TRUNNER_BPFTOOL := $(DEFAULT_BPFTOOL)
> +USE_BOOTSTRAP := "bootstrap"
> +endif
>
> $(OUTPUT)/runqslower: $(BPFOBJ) | $(DEFAULT_BPFTOOL) $(RUNQSLOWER_OUTPUT)
> $(Q)$(MAKE) $(submake_extras) -C $(TOOLSDIR)/bpf/runqslower \
...
-TEST_GEN_PROGS_EXTENDED += $(DEFAULT_BPFTOOL)
+TEST_GEN_PROGS_EXTENDED += $(TRUNNER_BPFTOOL)
Ensure the target arch bpftool is copied into the kselftest_bpf_install
dir by selftests/lib.mk instead of always the default/host version.
Thanks,
Zach
> @@ -255,6 +264,18 @@ $(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
> LIBBPF_DESTDIR=$(HOST_SCRATCH_DIR)/ \
> prefix= DESTDIR=$(HOST_SCRATCH_DIR)/ install-bin
>
> +ifneq ($(CROSS_COMPILE),)
> +$(CROSS_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \
> + $(BPFOBJ) | $(BUILD_DIR)/bpftool
> + $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \
> + ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) \
> + EXTRA_CFLAGS='-g -O0' \
> + OUTPUT=$(BUILD_DIR)/bpftool/ \
> + LIBBPF_OUTPUT=$(BUILD_DIR)/libbpf/ \
> + LIBBPF_DESTDIR=$(SCRATCH_DIR)/ \
> + prefix= DESTDIR=$(SCRATCH_DIR)/ install-bin
> +endif
> +
> all: docs
>
> docs:
> @@ -518,11 +539,12 @@ endif
> $(OUTPUT)/$(TRUNNER_BINARY): $(TRUNNER_TESAT_OBJS) \
> $(TRUNNER_EXTRA_OBJS) $$(BPFOBJ) \
> $(RESOLVE_BTFIDS) related to \
> + $(TRUNNER_BPFTOOL) \
> | $(TRUNNER_BINARY)-extras
> $$(call msg,BINARY,,$$@)
> $(Q)$$(CC) $$(CFLAGS) $$(filter %.a %.o,$$^) $$(LDLIBS) -o $$@
> $(Q)$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.bpf.o $$@
> - $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \
> + $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/$(USE_BOOTSTRAP)/bpftool \> $(OUTPUT)/$(if $2,$2/)bpftool
>
> endef
>
> base-commit: 06744f24696e1e7598412c3df61a538b57ebec22
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH bpf-next] selftests/bpf: Cross-compile bpftool
2023-02-14 8:57 ` Zachary Leaf
@ 2023-02-14 9:41 ` Björn Töpel
0 siblings, 0 replies; 8+ messages in thread
From: Björn Töpel @ 2023-02-14 9:41 UTC (permalink / raw)
To: Zachary Leaf, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, bpf, netdev
Cc: Björn Töpel, Jean-Philippe Brucker, linux-kselftest,
Anders Roxell, linux-morello, Quentin Monnet
Zachary Leaf <zachary.leaf@arm.com> writes:
> Hi Bjorn,
>
> Thanks for the patch, I've tested it and it works for me.
Good!
> I have a minor suggestion but otherwise happy to see this getting fixed.
>
> ...
>
>
> -TEST_GEN_PROGS_EXTENDED += $(DEFAULT_BPFTOOL)
> +TEST_GEN_PROGS_EXTENDED += $(TRUNNER_BPFTOOL)
>
> Ensure the target arch bpftool is copied into the kselftest_bpf_install
> dir by selftests/lib.mk instead of always the default/host version.
Good one. I'll spin a v2, and also fix Quentin's "double-slash".
Björn
^ permalink raw reply [flat|nested] 8+ messages in thread