[RFC 0/2] perf build: Improve header handling for BPF skeleton cross-builds

[RFC 0/2] perf build: Improve header handling for BPF skeleton cross-builds

[hupu]

I am currently cross-compiling perf for ARM64. During the build of the
eBPF skeleton (compiled by Clang), the build fails due to missing header
files. The error message is as follows:

/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not
found
    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

Leo suggested resolving this issue by installing the following packages
on the host:

  $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
  $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
  $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

With these packages installed, the required header files can be found in
the following host paths (defined by the CLANG_SYS_INCLUDES variable),
and perf can indeed be built successfully:

  *** Makefile.perf:1203: ***
  CLANG_SYS_INCLUDES=-idirafter /usr/lib/llvm-18/lib/clang/18/include
  -idirafter /usr/local/include
  -idirafter /usr/bin/../lib/gcc-cross/aarch64-linux-gnu/14/../../../../aarch64-linux-gnu/include
  -idirafter /usr/include/aarch64-linux-gnu
  -idirafter /usr/include

However, I do not believe that relying on the host build environment is
the best approach, for several reasons:

a) These commands install UAPI header files on the host, especially
`linux-libc-dev-aarch64-cross` and `linux-libc-dev-arm64-cross`. These
headers originate from the kernel source tree’s `include/uapi/` and
`arch/arm64/include/uapi/` directories, and their versions are tied to
the *HOST* kernel version. If the target kernel version is different,
mismatches may cause compilation errors or even runtime failures.

b) Even if `perf` can be compiled and run successfully now, there is
no guarantee that the kernel source headers will always match the
host-installed UAPI headers as the upstream kernel evolves.

c) In scenarios where the host acts as a general build server and
needs to build multiple target kernel versions, it is not possible to
ensure that the host UAPI headers are compatible with all target
versions.

d) On the other hand, `CLANG_SYS_INCLUDES` does include host headers,
but it uses `-idirafter` instead of `-I`. This means the host headers
have lower priority. This change was introduced in commit a2af0f6b8ef7
("perf build: Add system include paths to BPF builds"); as noted in
the commit message, the preferred approach is to use kernel source
headers rather than potentially older ones from the system.


Therefore, although installing the corresponding packages on the host
can resolve the immediate build issue, I do not think this should be
considered the only solution, nor should it have the highest priority.

Instead, I think the header search order should be as follows.

- Prefer kernel source headers
  [RFC 2/2] perf build: Prefer kernel source headers for BPF skeletons
  With this approach, even as the kernel version evolves, the headers
  used during the build will always match the target kernel version.

- Allow users to specify header search paths matching the target kernel
  version (eg. via `EXTRA_BPF_FLAGS`)
  [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS
  With this approach, users can explicitly specify the appropriate
  header search paths via EXTRA_BPF_FLAGS.

- Fall back to the host build environment only if necessary
  Reliance on the host build environment should be the last resort.

In summary, while relying on the host build environment can address the
current build issue, I believe it is not the optimal solution.

hupu (2):
  perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS
  perf build: Prefer kernel source headers for BPF skeletons

 tools/perf/Makefile.perf | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

-- 
2.43.0

[RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS

[hupu]

Add support for EXTRA_BPF_FLAGS in the eBPF skeleton build, allowing
users to pass additional Clang options such as --sysroot or custom
include paths when cross-compiling perf.

This is primarily intended for cross-build scenarios where the default
host include paths do not match the target kernel version.

Example usage:
    make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Change history:
  v5:
    - Update commit message
  v4:
    - Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command
      instead of modifying CLANG_OPTIONS
  v3:
    - Move the variable description comment to the top of Makefile.perf
    - Update commit message
  v2:
    - Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
    - Update commit message
  v1:
    - Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

Signed-off-by: hupu <hupu.gm@gmail.com>
---
 tools/perf/Makefile.perf | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index b3f481a626af..8b8647fd4f41 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak
 #
 # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds.
 #
+# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for
+# cross-compiling BPF skeletons
+#
 # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated
 # EXTLIBS.
 #
@@ -1250,7 +1253,7 @@ endif
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
 	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
-	  $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
+	  $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
 	  -include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \
 	  -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@
 
-- 
2.43.0

[RFC 2/2] perf build: Prefer kernel source headers for BPF skeletons

[hupu]

When building eBPF skeletons with Clang, prefer header files provided by
the kernel source tree over headers installed on the host system.

While host-installed UAPI headers can be sufficient today, they are
tightly coupled to the host environment and may diverge from the target
kernel version over time as the kernel evolves. Relying on kernel source
headers helps ensure that the headers used during the build remain
consistent with the target kernel version and reduces dependence on the
host build environment.

This change adds the kernel source UAPI and generated header paths to
the BPF include list and places them before the system include paths,
while still keeping host headers (CLANG_SYS_INCLUDES) as a fallback.

Signed-off-by: hupu <hupu.gm@gmail.com>
---
 tools/perf/Makefile.perf | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 8b8647fd4f41..57c73ab98703 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -1203,7 +1203,13 @@ endif
 
 CLANG_OPTIONS = -Wall
 CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH))
-BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) $(CLANG_SYS_INCLUDES)
+
+KHDR_INCLUDES := -I$(abspath $(OUTPUT)../../usr/include) \
+-I$(abspath $(OUTPUT)../../arch/$(SRCARCH)/include/generated/uapi) \
+-I$(abspath $(OUTPUT)../../arch/$(SRCARCH)/include/uapi)
+
+BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) \
+		$(KHDR_INCLUDES) $(CLANG_SYS_INCLUDES)
 TOOLS_UAPI_INCLUDE := -I$(srctree)/tools/include/uapi
 
 ifneq ($(WERROR),0)
-- 
2.43.0

Re: [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS

[Namhyung Kim]

Hello,

On Tue, Dec 23, 2025 at 04:43:34PM +0800, hupu wrote:
> Add support for EXTRA_BPF_FLAGS in the eBPF skeleton build, allowing
> users to pass additional Clang options such as --sysroot or custom
> include paths when cross-compiling perf.
> 
> This is primarily intended for cross-build scenarios where the default
> host include paths do not match the target kernel version.
> 
> Example usage:
>     make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."
> 
> Change history:
>   v5:
>     - Update commit message
>   v4:
>     - Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command
>       instead of modifying CLANG_OPTIONS
>   v3:
>     - Move the variable description comment to the top of Makefile.perf
>     - Update commit message
>   v2:
>     - Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
>     - Update commit message
>   v1:
>     - Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options
> 
> Signed-off-by: hupu <hupu.gm@gmail.com>

Reviewed-by: Namhyung Kim <namhyung@kernel.org>

Thanks,
Namhyung

> ---
>  tools/perf/Makefile.perf | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
> index b3f481a626af..8b8647fd4f41 100644
> --- a/tools/perf/Makefile.perf
> +++ b/tools/perf/Makefile.perf
> @@ -35,6 +35,9 @@ include ../scripts/utilities.mak
>  #
>  # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds.
>  #
> +# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for
> +# cross-compiling BPF skeletons
> +#
>  # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated
>  # EXTLIBS.
>  #
> @@ -1250,7 +1253,7 @@ endif
>  $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
>  $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
>  	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
> -	  $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
> +	  $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
>  	  -include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \
>  	  -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@
>  
> -- 
> 2.43.0
> 

Re: [RFC 2/2] perf build: Prefer kernel source headers for BPF skeletons

[hupu]

Hi Maintainers,

Sorry for the noise.

This patch was sent nearly three weeks ago, but I have not received
any feedback yet.
I am resending this email as a gentle reminder in case it was missed.

Thank you very much for your time and review.

Best regards,
hupu

Re: [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS

[hupu]

On Wed, Dec 24, 2025 at 2:20 AM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Hello,
>
> On Tue, Dec 23, 2025 at 04:43:34PM +0800, hupu wrote:
> > Add support for EXTRA_BPF_FLAGS in the eBPF skeleton build, allowing
> > users to pass additional Clang options such as --sysroot or custom
> > include paths when cross-compiling perf.
> >
> > This is primarily intended for cross-build scenarios where the default
> > host include paths do not match the target kernel version.
> >
> > Example usage:
> >     make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."
> >
> > Change history:
> >   v5:
> >     - Update commit message
> >   v4:
> >     - Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command
> >       instead of modifying CLANG_OPTIONS
> >   v3:
> >     - Move the variable description comment to the top of Makefile.perf
> >     - Update commit message
> >   v2:
> >     - Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
> >     - Update commit message
> >   v1:
> >     - Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options
> >
> > Signed-off-by: hupu <hupu.gm@gmail.com>
>
> Reviewed-by: Namhyung Kim <namhyung@kernel.org>
>
> Thanks,
> Namhyung
>

Hi Namhyung,

Sorry to bother you.

I noticed that the patch adding support for EXTRA_BPF_FLAGS, which was
previously reviewed by you, does not seem to be included in the latest
released version. I just wanted to follow up and check on the current
status of this patch, in case it was missed or delayed for some
reason.

Thanks a lot for your time and review.

Thanks
hupu

Re: [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS

[Arnaldo Carvalho de Melo]

On Tue, Feb 10, 2026 at 01:54:49PM +0800, hupu wrote:
> On Wed, Dec 24, 2025 at 2:20 AM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > Hello,
> >
> > On Tue, Dec 23, 2025 at 04:43:34PM +0800, hupu wrote:
> > > Add support for EXTRA_BPF_FLAGS in the eBPF skeleton build, allowing
> > > users to pass additional Clang options such as --sysroot or custom
> > > include paths when cross-compiling perf.
> > >
> > > This is primarily intended for cross-build scenarios where the default
> > > host include paths do not match the target kernel version.
> > >
> > > Example usage:
> > >     make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."
> > >
> > > Change history:
> > >   v5:
> > >     - Update commit message
> > >   v4:
> > >     - Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command
> > >       instead of modifying CLANG_OPTIONS
> > >   v3:
> > >     - Move the variable description comment to the top of Makefile.perf
> > >     - Update commit message
> > >   v2:
> > >     - Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
> > >     - Update commit message
> > >   v1:
> > >     - Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options
> > >
> > > Signed-off-by: hupu <hupu.gm@gmail.com>
> >
> > Reviewed-by: Namhyung Kim <namhyung@kernel.org>
> >
> 
> Hi Namhyung,
> 
> Sorry to bother you.
> 
> I noticed that the patch adding support for EXTRA_BPF_FLAGS, which was
> previously reviewed by you, does not seem to be included in the latest
> released version. I just wanted to follow up and check on the current
> status of this patch, in case it was missed or delayed for some
> reason.
> 
> Thanks a lot for your time and review.

Applied this one,

- Arnaldo