Re: [RFC] perf build: Allow passing extra Clang flags via EXTRA_CLANG_FLAGS

[hupu <hupu.gm@gmail.com>]

Hi Maintainer,

I encountered some issues while cross-compiling perf (ARCH=arm64),
particularly when enabling the BPF option. During the build, Clang
fails to compile BPF programs due to missing header files. The
relevant error messages are as follows:


---------------------------------------
/usr/bin/ld: skipping incompatible
/home/hupu/work/tools/crosstool-ng/output/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libzstd.a
when searching for -lzstd
/usr/bin/ld: skipping incompatible
/home/hupu/work/tools/crosstool-ng/output/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/libc.a
when searching for -lc
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libcrypto.a(libcrypto-lib-dso_dlfcn.o):
in function `dlfcn_globallookup':
(.text+0x1f): warning: Using 'dlopen' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libcrypto.a(libcrypto-lib-bio_addr.o):
in function `BIO_lookup_ex':
(.text+0xdbc): warning: Using 'getaddrinfo' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libcrypto.a(libcrypto-lib-bio_sock.o):
in function `BIO_gethostbyname':
(.text+0x85): warning: Using 'gethostbyname' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_cgroup.bpf.o
In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from
/home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found
    In file included from
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:5
| #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~
7:
In file included from
/home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from
/home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found
    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found
    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~
1 error generated.
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o]
Error 1
make[3]: *** Waiting for unfinished jobs....
In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from
/home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found
    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~
1 error generated.
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_cgroup.bpf.o]
Error 1
1 error generated.
1 error generated.
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o]
Error 1
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o]
Error 1
make[2]: *** [Makefile.perf:289: sub-make] Error 2
make[1]: *** [Makefile:76: all] Error 2
make: *** [Makefile:93: perf] Error 2
---------------------------------------


This happens because the eBPF part of perf is compiled with Clang,
which by default searches only the host /usr/include/ paths for
headers, not the cross-compilation sysroot. As a result, it cannot
find the ARM64-specific asm/ioctl.h.

To address this issue, I wrote a patch that introduces a new make
variable, EXTRA_CLANG_FLAGS. This allows passing custom Clang options
via the command line during the BPF skeleton build, for example
--sysroot or custom include paths, without modifying Makefile.perf
itself.

Example usage:
  EXTRA_CLANG_FLAGS="--sysroot=$SYSROOT"
  make perf ARCH=arm64 EXTRA_CLANG_FLAGS="$EXTRA_CLANG_FLAGS"

I have tested this approach and it works. However, I am not sure if
this is the optimal solution, so I would like to discuss this further
with you and get your feedback.

Thanks,
hupu