Re: [PATCH v2] tools/sched_ext: Improve cross-compilation support in Makefile

[Andrea Righi <arighi@nvidia.com>]

On Tue, Apr 15, 2025 at 01:46:42PM +0800, yangsonghua wrote:
> Modify the tools/sched_ext/Makefile to better handle cross-compilation
> environments by:
> 
> 1. Adjusted `HOST_OUTPUT_DIR` to be relative to `$(OBJ_DIR)`, ensuring
>    correct path handling during host tool building when cross-compile
>    (HOST_OUTPUT_DIR now points to $(OBJ_DIR)/host-tools)
> 2. Properly propagate CROSS_COMPILE to libbpf sub-make invocation
> 3. Add missing $(HOST_BPFOBJ) build rule with proper host toolchain flags
>    (ARCH=, CROSS_COMPILE=, explicit HOSTCC/HOSTLD)
> 4. Consistently quote $(HOSTCC) in bpftool build rule
> 
> The changes ensure proper cross-compilation behavior while maintaining
> backward compatibility with native builds. Host tools are now correctly
> built with the host toolchain while target binaries use the cross-toolchain.
> 
> Signed-off-by: yangsonghua <yangsonghua@lixiang.com>

It makes things better with this applied, at least it starts the cross
build, but I'm still getting troubles with libelf and zlib:

$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64
...
  LINK    /home/arighi/src/linux/tools/sched_ext/build/obj/libbpf/libbpf.so.1.6.0
/usr/lib/gcc-cross/aarch64-linux-gnu/14/../../../../aarch64-linux-gnu/bin/ld: cannot find -lelf: No such file or directory
/usr/lib/gcc-cross/aarch64-linux-gnu/14/../../../../aarch64-linux-gnu/bin/ld: cannot find -lz: No such file or directory

Which distro are you using and what command are you running exactly to test
the cross build?

Thanks,
-Andrea

> 
> ------
> v2:
>   - keep HOST_BUILD_DIR and LDFLAGS unchanged
>   - change the title prefix "sched_ext" to "tools/sched_ext"
> 
> v1:
>   - https://lore.kernel.org/lkml/20250414081436.2574882-1-yangsonghua@lixiang.com/
> ---
>  tools/sched_ext/Makefile | 19 +++++++++++++++----
>  1 file changed, 15 insertions(+), 4 deletions(-)
> 
> diff --git a/tools/sched_ext/Makefile b/tools/sched_ext/Makefile
> index ca3815e572d8..eb85f3762a1f 100644
> --- a/tools/sched_ext/Makefile
> +++ b/tools/sched_ext/Makefile
> @@ -62,7 +62,7 @@ BINDIR := $(OUTPUT_DIR)/bin
>  BPFOBJ := $(BPFOBJ_DIR)/libbpf.a
>  ifneq ($(CROSS_COMPILE),)
>  HOST_BUILD_DIR		:= $(OBJ_DIR)/host
> -HOST_OUTPUT_DIR	:= host-tools
> +HOST_OUTPUT_DIR	:= $(OBJ_DIR)/host-tools
>  HOST_INCLUDE_DIR	:= $(HOST_OUTPUT_DIR)/include
>  else
>  HOST_BUILD_DIR		:= $(OBJ_DIR)
> @@ -136,14 +136,25 @@ $(MAKE_DIRS):
>  $(BPFOBJ): $(wildcard $(BPFDIR)/*.[ch] $(BPFDIR)/Makefile)			\
>  	   $(APIDIR)/linux/bpf.h						\
>  	   | $(OBJ_DIR)/libbpf
> -	$(Q)$(MAKE) $(submake_extras) -C $(BPFDIR) OUTPUT=$(OBJ_DIR)/libbpf/	\
> +	$(Q)$(MAKE) $(submake_extras) CROSS_COMPILE=$(CROSS_COMPILE) 		\
> +		    -C $(BPFDIR) OUTPUT=$(OBJ_DIR)/libbpf/			\
>  		    EXTRA_CFLAGS='-g -O0 -fPIC'					\
> +		    LDFLAGS="$(LDFLAGS)"					\
>  		    DESTDIR=$(OUTPUT_DIR) prefix= all install_headers
>  
> +$(HOST_BPFOBJ): $(wildcard $(BPFDIR)/*.[ch] $(BPFDIR)/Makefile)		\
> +	   $(APIDIR)/linux/bpf.h						\
> +	   | $(HOST_BUILD_DIR)/libbpf
> +	$(Q)$(MAKE) $(submake_extras) -C $(BPFDIR) 				\
> +		    OUTPUT=$(HOST_BUILD_DIR)/libbpf/				\
> +		    ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD=$(HOSTLD)		\
> +		    EXTRA_CFLAGS='-g -O0 -fPIC'					\
> +		    DESTDIR=$(HOST_OUTPUT_DIR) prefix= all install_headers
> +
>  $(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile)	\
>  		    $(HOST_BPFOBJ) | $(HOST_BUILD_DIR)/bpftool
>  	$(Q)$(MAKE) $(submake_extras)  -C $(BPFTOOLDIR)				\
> -		    ARCH= CROSS_COMPILE= CC=$(HOSTCC) LD=$(HOSTLD)		\
> +		    ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD=$(HOSTLD)		\
>  		    EXTRA_CFLAGS='-g -O0'					\
>  		    OUTPUT=$(HOST_BUILD_DIR)/bpftool/				\
>  		    LIBBPF_OUTPUT=$(HOST_BUILD_DIR)/libbpf/			\
> @@ -185,7 +196,7 @@ $(addprefix $(BINDIR)/,$(c-sched-targets)): \
>  		$(SCX_COMMON_DEPS)
>  	$(eval sched=$(notdir $@))
>  	$(CC) $(CFLAGS) -c $(sched).c -o $(SCXOBJ_DIR)/$(sched).o
> -	$(CC) -o $@ $(SCXOBJ_DIR)/$(sched).o $(HOST_BPFOBJ) $(LDFLAGS)
> +	$(CC) -o $@ $(SCXOBJ_DIR)/$(sched).o $(BPFOBJ) $(LDFLAGS)
>  
>  $(c-sched-targets): %: $(BINDIR)/%
>  
> -- 
> 2.25.1
>