[linux-next-20250324]/tool/bpf/bpftool fails to complie on linux-next-20250324

[linux-next-20250324]/tool/bpf/bpftool fails to complie on linux-next-20250324

[Venkat Rao Bagalkote]

Greetings!!!


bpftool fails to complie on linux-next-20250324 repo.


Error:

make: *** No rule to make target 'bpftool', needed by 
'/home/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h'. Stop.
make: *** Waiting for unfinished jobs.....


Git bisect points to commit: 8a635c3856ddb74ed3fe7c856b271cdfeb65f293 as 
first bad commit.

Bisect log:

git bisect start
# status: waiting for both good and bad commits
# good: [4701f33a10702d5fc577c32434eb62adde0a1ae1] Linux 6.14-rc7
git bisect good 4701f33a10702d5fc577c32434eb62adde0a1ae1
# status: waiting for bad commit, 1 good commit known
# bad: [882a18c2c14fc79adb30fe57a9758283aa20efaa] Add linux-next 
specific files for 20250324
git bisect bad 882a18c2c14fc79adb30fe57a9758283aa20efaa
# good: [36ad536dbad8e29a1fdb7a8760a9c4fcb0dcf7cb] Merge branch 
'for-next' of 
git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next.git
git bisect good 36ad536dbad8e29a1fdb7a8760a9c4fcb0dcf7cb
# good: [96c123361d8e32f6012aa449eed27147979af27e] Merge branch 'next' 
of git://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd.git
git bisect good 96c123361d8e32f6012aa449eed27147979af27e
# bad: [b9fc57d1f74797e7b25c779671c03192a81feb1a] Merge branch 
'usb-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git
git bisect bad b9fc57d1f74797e7b25c779671c03192a81feb1a
# good: [1da0a3d00734bf365f53480a7ffb4361fd61e6d5] Merge branch 'master' 
of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
git bisect good 1da0a3d00734bf365f53480a7ffb4361fd61e6d5
# bad: [4541ffab99f8b7ddadb367c73f28ea1fe70f2f97] Merge branch 'next' of 
git://git.kernel.org/pub/scm/virt/kvm/kvm.git
git bisect bad 4541ffab99f8b7ddadb367c73f28ea1fe70f2f97
# good: [361da275e5ce98bbab5f6990d02eb9709742d703] Merge branch 
'kvm-nvmx-and-vm-teardown' into HEAD
git bisect good 361da275e5ce98bbab5f6990d02eb9709742d703
# bad: [28b4c36e59ccfd4e38eaf804b292b3c5b2287900] Merge branch 
'for-next' of 
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git
git bisect bad 28b4c36e59ccfd4e38eaf804b292b3c5b2287900
# good: [2ec5357274fdbe8d48d13d33a1b0e367bcadb85a] Merge sorttable/for-next
git bisect good 2ec5357274fdbe8d48d13d33a1b0e367bcadb85a
# good: [af1a78613133542583c9a9875c824678a3c3a145] Merge branch 
'edac-drivers' into edac-for-next
git bisect good af1a78613133542583c9a9875c824678a3c3a145
# good: [2325ccf7b99fa8e1e95c3ce8a205e170d244b062] Merge branch 
'edac-for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras.git
git bisect good 2325ccf7b99fa8e1e95c3ce8a205e170d244b062
# bad: [18923806b1291102cad3a6b713006c7e7f563534] rtla/timerlat_top: 
Move divisor to update
git bisect bad 18923806b1291102cad3a6b713006c7e7f563534
# bad: [9dc3766ed07c95c9a77fa98dcbc83dcb7f49df3d] rtla: Add optional 
dependency on BPF tooling
git bisect bad 9dc3766ed07c95c9a77fa98dcbc83dcb7f49df3d
# bad: [8a635c3856ddb74ed3fe7c856b271cdfeb65f293] tools/build: Add 
bpftool-skeletons feature test
git bisect bad 8a635c3856ddb74ed3fe7c856b271cdfeb65f293
# good: [6fa5e3a87cd7838453be66c3a69c2236a1680504] rtla/timerlat: Unify 
params struct
git bisect good 6fa5e3a87cd7838453be66c3a69c2236a1680504
# first bad commit: [8a635c3856ddb74ed3fe7c856b271cdfeb65f293] 
tools/build: Add bpftool-skeletons feature test


If you happen to fix this, please add below tag.


Reported-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com>


Regards,

Venkat.

Re: [linux-next-20250324]/tool/bpf/bpftool fails to complie on linux-next-20250324

[Quentin Monnet]

2025-03-25 16:02 UTC+0530 ~ Venkat Rao Bagalkote <venkat88@linux.ibm.com>
> Greetings!!!
> 
> 
> bpftool fails to complie on linux-next-20250324 repo.
> 
> 
> Error:
> 
> make: *** No rule to make target 'bpftool', needed by '/home/linux/
> tools/testing/selftests/bpf/tools/include/vmlinux.h'. Stop.
> make: *** Waiting for unfinished jobs.....


Thanks! Would be great to have a bit more context on the error (and on
how to reproduce) for next time. Bpftool itself seems to compile fine,
the error shows that it's building it from the context of the selftests
that seems broken.


> Git bisect points to commit: 8a635c3856ddb74ed3fe7c856b271cdfeb65f293 as
> first bad commit.

Thank you Venkat for the bisect!

On a quick look, that commit introduced a definition for BPFTOOL in
tools/scripts/Makefile.include:

	diff --git a/tools/scripts/Makefile.include .../Makefile.include
	index 0aa4005017c7..71bbe52721b3 100644
	--- a/tools/scripts/Makefile.include
	+++ b/tools/scripts/Makefile.include
	@@ -91,6 +91,9 @@ LLVM_CONFIG	?= llvm-config
	 LLVM_OBJCOPY	?= llvm-objcopy
	 LLVM_STRIP	?= llvm-strip
	
	+# Some tools require bpftool
	+BPFTOOL		?= bpftool
	+
	 ifeq ($(CC_NO_CLANG), 1)
	 EXTRA_WARNINGS += -Wstrict-aliasing=3

But several utilities or selftests under tools/ include
tools/scripts/Makefile.include _and_ use their own version of the
$(BPFTOOL) variable, often assigning only if unset, for example in
tools/testing/selftests/bpf/Makefile:

	BPFTOOL ?= $(DEFAULT_BPFTOOL)

My guess is that the new definition from Makefile.include overrides this
with simply "bpftool" as a value, and the Makefile fails to build it as
a result.

If I guessed correctly, one workaround would be to rename the variable
in Makefile.include (and in whatever Makefile now relies on it) into
something that is not used in the other Makefiles, for example
BPFTOOL_BINARY.

Please copy the BPF mailing list on changes impacting BPF tooling (or
for BPF-related patchsets in general).

Thanks,
Quentin

Re: [linux-next-20250324]/tool/bpf/bpftool fails to complie on linux-next-20250324

[Saket Kumar Bhaskar]

On Tue, Mar 25, 2025 at 11:09:24AM +0000, Quentin Monnet wrote:
> 2025-03-25 16:02 UTC+0530 ~ Venkat Rao Bagalkote <venkat88@linux.ibm.com>
> > Greetings!!!
> > 
> > 
> > bpftool fails to complie on linux-next-20250324 repo.
> > 
> > 
> > Error:
> > 
> > make: *** No rule to make target 'bpftool', needed by '/home/linux/
> > tools/testing/selftests/bpf/tools/include/vmlinux.h'. Stop.
> > make: *** Waiting for unfinished jobs.....
> 
> 
> Thanks! Would be great to have a bit more context on the error (and on
> how to reproduce) for next time. Bpftool itself seems to compile fine,
> the error shows that it's building it from the context of the selftests
> that seems broken.
> 
> 
Yes, selftest build for BPF fails.
## pwd
/linux/tools/testing/selftests/bpf

# make -j 33

make: *** No rule to make target 'bpftool', needed by '/home/upstreamci/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h'.  Stop.
make: *** Waiting for unfinished jobs....

> > Git bisect points to commit: 8a635c3856ddb74ed3fe7c856b271cdfeb65f293 as
> > first bad commit.
> 
> Thank you Venkat for the bisect!
> 
> On a quick look, that commit introduced a definition for BPFTOOL in
> tools/scripts/Makefile.include:
> 
> 	diff --git a/tools/scripts/Makefile.include .../Makefile.include
> 	index 0aa4005017c7..71bbe52721b3 100644
> 	--- a/tools/scripts/Makefile.include
> 	+++ b/tools/scripts/Makefile.include
> 	@@ -91,6 +91,9 @@ LLVM_CONFIG	?= llvm-config
> 	 LLVM_OBJCOPY	?= llvm-objcopy
> 	 LLVM_STRIP	?= llvm-strip
> 	
> 	+# Some tools require bpftool
> 	+BPFTOOL		?= bpftool
> 	+
> 	 ifeq ($(CC_NO_CLANG), 1)
> 	 EXTRA_WARNINGS += -Wstrict-aliasing=3
> 
> But several utilities or selftests under tools/ include
> tools/scripts/Makefile.include _and_ use their own version of the
> $(BPFTOOL) variable, often assigning only if unset, for example in
> tools/testing/selftests/bpf/Makefile:
> 
> 	BPFTOOL ?= $(DEFAULT_BPFTOOL)
> 
> My guess is that the new definition from Makefile.include overrides this
> with simply "bpftool" as a value, and the Makefile fails to build it as
> a result.
> 
> If I guessed correctly, one workaround would be to rename the variable
> in Makefile.include (and in whatever Makefile now relies on it) into
> something that is not used in the other Makefiles, for example
> BPFTOOL_BINARY.
> 
> Please copy the BPF mailing list on changes impacting BPF tooling (or
> for BPF-related patchsets in general).
> 
> Thanks,
> Quentin
Yes you are right that the new definition from Makefile.include overrides this
with simply "bpftool" as a value, and the Makefile in bpf selftest fails to 
build it as a result.

But the main cause is that it is not able to locate the bpftool binary.
So, is it good idea to both rename this variable in Makefile.include and 
use:

BPFTOOL ?= /usr/sbin/bpftool

This is the link to patch that is impacting: 
https://lore.kernel.org/all/20250218145859.27762-3-tglozar@redhat.com/

Thanks,
Saket

Re: [linux-next-20250324]/tool/bpf/bpftool fails to complie on linux-next-20250324

[Quentin Monnet]

2025-03-25 17:14 UTC+0530 ~ Saket Kumar Bhaskar <skb99@linux.ibm.com>
> On Tue, Mar 25, 2025 at 11:09:24AM +0000, Quentin Monnet wrote:
>> 2025-03-25 16:02 UTC+0530 ~ Venkat Rao Bagalkote <venkat88@linux.ibm.com>
>>> Greetings!!!
>>>
>>>
>>> bpftool fails to complie on linux-next-20250324 repo.
>>>
>>>
>>> Error:
>>>
>>> make: *** No rule to make target 'bpftool', needed by '/home/linux/
>>> tools/testing/selftests/bpf/tools/include/vmlinux.h'. Stop.
>>> make: *** Waiting for unfinished jobs.....
>>
>>
>> Thanks! Would be great to have a bit more context on the error (and on
>> how to reproduce) for next time. Bpftool itself seems to compile fine,
>> the error shows that it's building it from the context of the selftests
>> that seems broken.
>>
>>
> Yes, selftest build for BPF fails.
> ## pwd
> /linux/tools/testing/selftests/bpf
> 
> # make -j 33
> 
> make: *** No rule to make target 'bpftool', needed by '/home/upstreamci/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h'.  Stop.
> make: *** Waiting for unfinished jobs....
> 
>>> Git bisect points to commit: 8a635c3856ddb74ed3fe7c856b271cdfeb65f293 as
>>> first bad commit.
>>
>> Thank you Venkat for the bisect!
>>
>> On a quick look, that commit introduced a definition for BPFTOOL in
>> tools/scripts/Makefile.include:
>>
>> 	diff --git a/tools/scripts/Makefile.include .../Makefile.include
>> 	index 0aa4005017c7..71bbe52721b3 100644
>> 	--- a/tools/scripts/Makefile.include
>> 	+++ b/tools/scripts/Makefile.include
>> 	@@ -91,6 +91,9 @@ LLVM_CONFIG	?= llvm-config
>> 	 LLVM_OBJCOPY	?= llvm-objcopy
>> 	 LLVM_STRIP	?= llvm-strip
>> 	
>> 	+# Some tools require bpftool
>> 	+BPFTOOL		?= bpftool
>> 	+
>> 	 ifeq ($(CC_NO_CLANG), 1)
>> 	 EXTRA_WARNINGS += -Wstrict-aliasing=3
>>
>> But several utilities or selftests under tools/ include
>> tools/scripts/Makefile.include _and_ use their own version of the
>> $(BPFTOOL) variable, often assigning only if unset, for example in
>> tools/testing/selftests/bpf/Makefile:
>>
>> 	BPFTOOL ?= $(DEFAULT_BPFTOOL)
>>
>> My guess is that the new definition from Makefile.include overrides this
>> with simply "bpftool" as a value, and the Makefile fails to build it as
>> a result.
>>
>> If I guessed correctly, one workaround would be to rename the variable
>> in Makefile.include (and in whatever Makefile now relies on it) into
>> something that is not used in the other Makefiles, for example
>> BPFTOOL_BINARY.
>>
>> Please copy the BPF mailing list on changes impacting BPF tooling (or
>> for BPF-related patchsets in general).
>>
>> Thanks,
>> Quentin
> Yes you are right that the new definition from Makefile.include overrides this
> with simply "bpftool" as a value, and the Makefile in bpf selftest fails to 
> build it as a result.
> 
> But the main cause is that it is not able to locate the bpftool binary.

I'm not sure I follow. What component is not able to locate the binary?

If you talk about the BPF selftests, I believe they only fail to locate
it because of the collision on the $(BPFTOOL) variable. Selftests'
Makefile was able to find the binary before that commit, so there should
be no need to change the path to the binary.

If you talk about tools/tracing/rtla/Makefile failing to locate bpftool,
it's another matter. As far as I understand, the RTLA Makefile assumes
that bpftool is available from $PATH, this is why the commit introduced
a probe in tools/build/feature: to ensure that bpftool is installed and
available. So here again, I don't see the motivation for changing the
path to the binary (And how do you know it's /usr/sbin/bpftool anyway?
Some users have it under /usr/local/sbin/, for example). If the intent
were to compile a bootstrap bpftool to make sure that it's available
instead then it should replicate what other BPF utilities or selftests
do, and get rid of the probe. But the commit description for
8a635c3856dd indicates that RTLA folks prefer not to compile bpftool and
rely on it being installed.

Quentin

Re: [linux-next-20250324]/tool/bpf/bpftool fails to complie on linux-next-20250324

[Tomas Glozar]

Hello Quentin, Venkat, Saket,

Thanks for looking into this.

út 25. 3. 2025 v 13:12 odesílatel Quentin Monnet <qmo@kernel.org> napsal:
> If you talk about tools/tracing/rtla/Makefile failing to locate bpftool,
> it's another matter. As far as I understand, the RTLA Makefile assumes
> that bpftool is available from $PATH, this is why the commit introduced
> a probe in tools/build/feature: to ensure that bpftool is installed and
> available. So here again, I don't see the motivation for changing the
> path to the binary (And how do you know it's /usr/sbin/bpftool anyway?
> Some users have it under /usr/local/sbin/, for example). If the intent
> were to compile a bootstrap bpftool to make sure that it's available
> instead then it should replicate what other BPF utilities or selftests
> do, and get rid of the probe. But the commit description for
> 8a635c3856dd indicates that RTLA folks prefer not to compile bpftool and
> rely on it being installed.

Yes, that is correct. The reason why I opted to use the system bpftool
is that bpftool itself has a lot of dependencies, and they would have
to be available at the time of building RTLA. Since RTLA only requires
basic bpftool skeleton generation, and the only "special" feature it
uses is CO-RE which is already quite old now, I don't expect the build
to fail with system bpftool, so I chose to use that to make both the
build dependencies and the RTLA Makefiles simpler.

My commits sets BPFTOOL to bpftool since otherwise, the feature check
would fail, as BPFTOOL wouldn't be defined, since it is not passed to
the feature detection make call. I observed we are doing the same for
Clang and the LLVM toolchain in tools/scripts/Makefile.include;
obviously, there is no problem there, since neither of these are
in-kernel.

Shouldn't the selftests always test the in-tree bpftool instead of the
system one? Let's say there is a stray BPFTOOL environmental variable.
In that case, the tests will give incorrect, possibly false negative
results, if the user is expecting selftests to test what is in the
kernel tree. If it is intended to also be able to test with another
version of bpftool, we can work around the problem by removing the
BPFTOOL definition from tools/scripts/Makefile.include and exporting
it from the rtla Makefiles instead, to make sure the feature tests see
it. The problem with that is, obviously, that future users of the
bpftool feature check would have to do the same, or they would always
fail, unless the user sets BPFTOOL as an environment variable
themselves.

Tomas

Re: [linux-next-20250324]/tool/bpf/bpftool fails to complie on linux-next-20250324

[Tomas Glozar]

út 25. 3. 2025 v 15:59 odesílatel Tomas Glozar <tglozar@redhat.com> napsal:
> Shouldn't the selftests always test the in-tree bpftool instead of the
> system one? Let's say there is a stray BPFTOOL environmental variable.
> In that case, the tests will give incorrect, possibly false negative
> results, if the user is expecting selftests to test what is in the
> kernel tree. If it is intended to also be able to test with another
> version of bpftool, we can work around the problem by removing the
> BPFTOOL definition from tools/scripts/Makefile.include and exporting
> it from the rtla Makefiles instead, to make sure the feature tests see
> it. The problem with that is, obviously, that future users of the
> bpftool feature check would have to do the same, or they would always
> fail, unless the user sets BPFTOOL as an environment variable
> themselves.

Or the selftests and other users could use another variable, like
BPFTOOL_TEST or BPFTOOL_INTERNAL. Not sure what you BPF folks think
about that. I believe assuming BPFTOOL refers to the system bpftool
(just like it does for all the other tools) is quite reasonable.

Tomas

Re: [linux-next-20250324]/tool/bpf/bpftool fails to complie on linux-next-20250324

[Quentin Monnet]

> My commits sets BPFTOOL to bpftool since otherwise, the feature check
> would fail, as BPFTOOL wouldn't be defined, since it is not passed to
> the feature detection make call.


Sorry I don't understand the issue, why not simply rename the variable
that you introduced in tools/build/feature/Makefile at the same time, as
well? That should solve it, no? This way you don't have to export it
from the rtla Makefiles. Or am I missing something?


2025-03-25 16:09 UTC+0100 ~ Tomas Glozar <tglozar@redhat.com>
> út 25. 3. 2025 v 15:59 odesílatel Tomas Glozar <tglozar@redhat.com> napsal:
>> Shouldn't the selftests always test the in-tree bpftool instead of the
>> system one? Let's say there is a stray BPFTOOL environmental variable.
>> In that case, the tests will give incorrect, possibly false negative
>> results, if the user is expecting selftests to test what is in the
>> kernel tree. If it is intended to also be able to test with another


I think this was the intent.


>> version of bpftool, we can work around the problem by removing the
>> BPFTOOL definition from tools/scripts/Makefile.include and exporting
>> it from the rtla Makefiles instead, to make sure the feature tests see
>> it. The problem with that is, obviously, that future users of the
>> bpftool feature check would have to do the same, or they would always
>> fail, unless the user sets BPFTOOL as an environment variable
>> themselves.
> 
> Or the selftests and other users could use another variable, like
> BPFTOOL_TEST or BPFTOOL_INTERNAL. Not sure what you BPF folks think
> about that. I believe assuming BPFTOOL refers to the system bpftool
> (just like it does for all the other tools) is quite reasonable.


The variable name needs to change either for rtla + probe, or for all
BPF utilities relying on it, indeed. As far as I can see, this is the
sched_ext and runqslower utilities as well as the selftests for bpf,
sched_ext, and hid. I'd argue that the variable has been in use in the
Makefiles for these tools and selftests for a while, and renaming it
might produce errors for anyone already using it to pass a specfic
version of bpftool to try.


> The reason why I opted to use the system bpftool is that bpftool
> itself has a lot of dependencies


Note: Not that many dependencies, most of them are optional. For
bootstrap bpftool we pass -lelf, -lz, sometimes -lzstd.

Quentin

Re: [linux-next-20250324]/tool/bpf/bpftool fails to complie on linux-next-20250324

[Tomas Glozar]

út 25. 3. 2025 v 16:27 odesílatel Quentin Monnet <qmo@kernel.org> napsal:
> Sorry I don't understand the issue, why not simply rename the variable
> that you introduced in tools/build/feature/Makefile at the same time, as
> well? That should solve it, no? This way you don't have to export it
> from the rtla Makefiles. Or am I missing something?
>

If I set BPFTOOL in the rtla makefiles, then it won't propagate to the
feature check, unless exported. I observed feature testing of clang
works, because CLANG is set in tools/scripts/Makefile.include, and did
the same thing for BPFTOOL.

> I think this was the intent.

I see.

> The variable name needs to change either for rtla + probe, or for all
> BPF utilities relying on it, indeed. As far as I can see, this is the
> sched_ext and runqslower utilities as well as the selftests for bpf,
> sched_ext, and hid. I'd argue that the variable has been in use in the
> Makefiles for these tools and selftests for a while, and renaming it
> might produce errors for anyone already using it to pass a specfic
> version of bpftool to try.

That sounds much better than renaming the existing BPFTOOL variable,
thanks for the suggestion. I will send a patch tomorrow and give you a
Suggested-by.

> Note: Not that many dependencies, most of them are optional. For
> bootstrap bpftool we pass -lelf, -lz, sometimes -lzstd.

Noted. I must have been thinking of the entire long list of
dependencies in tools/perf/Makefile.config, completely unrelated to
this. Sorry for the confusion.

Tomas