Re: [RFC] running bpftool build tests in CI

[Ihor Solodrai <ihor.solodrai@linux.dev>]

On 2026-08-20 1:34 a.m., Alexis Lothoré wrote:
> Hi Ihor,
> thanks for the feedback
> 
> On Wed Aug 19, 2026 at 10:46 PM CEST, Ihor Solodrai wrote:
>> On 8/19/26 12:59 PM, Alexis Lothoré wrote:
>>> Hi,
>>> as part of the cleanup/automation effort in the BPF selfests directory,
>>> I am now taking a look at test_bpftool_build.sh, which ensures that the
>>> different supported ways of building bpftool work correctly. As this
>>> script does not really exercize anything at runtime but rather at build
>>> time, I'd like to propose to introduce a dedicated step in the CI
>>> automation that already builds and run selftests. I have opened two PRs
>>> in kernel-patches vmtest ([0]) and libbpf/ci ([1]), hoping I am not
>>> confusing which code should go where between the different repositories
>>> kept in sync with each other. I have also opened a dummy PR on
>>> kernel-patches/bpf ([2], not to be merged) that shows how this test
>>> would look like in CI.
>>>
>>> This really is a RFC, as not all tests from test_bpftool_build.sh are
>>> being executed: I suspect those based on .config to be currently broken,
>>> but that can be handled as a second step, depending on the chosen
>>> direction.
>>>
>>> Any comment welcome !
>>
>> Hi Alexis, thank you for working on this.
>>
>> I am a little confused about what are we trying to test here and
>> why. Maybe you could explain.
> 
> Sure, I may have been a bit light on this. The test_bpftool_build.sh
> located in tools/testing/selftests/bpf focuses on the various ways of
> building bpftool:
> - through kbuild (eg: make tools/bpf)
> - by changing make execution dir (eg: make -C tools/bpf/bpftool)
> - by running the tools/ main makefile (eg: cd tools && make bpf)
> - by running the bpftool main makefile (eg: cd tools/bpf/bpftool &&
>    make)
> 
> This listing is also cross-tested with output path configuration,
> testing both O=<output_dir> and OUTPUT=<output_dir>.

Right. That's what I meant by "it's testing makefile infra".  It's
checking whether any of the listed ways for initiating the build is
broken.

I agree that adding tests of this kind to CI is a good idea. My point
is it's worth expanding the coverage beyond the bpftool.

There was a patchset with fixes for out-of-tree build of selftests
recently, for example [1]. The tests would help to catch this earlier.

[1] 
https://lore.kernel.org/bpf/20260728-selftests-bpf_oot-v1-0-05feb15d94db@marliere.net/

> 
> Will be included in the relevant commits if this RFC goes further in
> this direction.
> 
>> Some variants of bpftool build are CI-exercised regularly:
>>    * selftests/bpf directly depend on bpftool, it's built and is used
>>      as part of the main "test_progs*" suite
>>    * bpftool has a standalone github mirror with it's own CI pipeline:
>>      https://github.com/libbpf/bpftool
>>
>> Looking at test_bpftool_build.sh (written 6y ago btw), it seems to be
>> more of a Makefile infra test than bpftool test specifically.
>>
>> Given how complicated some in-tree tools/selftests/ Makefiles are [1],
>> running infra tests like this would be nice. But then maybe we should
>> develop them beyond test_bpftool_build.sh?
>>
>> I briefly skimmed over the CI PRs you've submitted, and one thing I
>> would say is that we don't want a "bpftool build" test to block the
>> kernel build when it fails. It either needs to be a separate job, or a
>> step that is allowed to fail.
>>
>> Vineet recently did a similar thing, where the gcc-bpf selftests build
>> was interleaved with the kernel build and I nacked it with the same
>> justification [2].
> 
> Understood, thanks for the pointers. I see that Vineet came up with a
> new revision that eventually got merged, moving gcc-bpf as a dedicated
> job in kernel-build-test.yml instead of kernel-build.yml (and so, a
> failure on gcc-bpf does not mark kernel build as failed) I guess I can
> replicate this (also, making sure to add this run_tests toggle, maybe
> ?). However, I feel like it would not make much sense to have a build
> and a test part, with artifacts going from the former to the latter, as
> the build step actually _is_ the test. Also, the only needed artifact to
> run the build is the kernel source tree (we don't need any vmlinux or
> .config, at least for now)
> 
> Would it be ok if I try it this way ?

I think if the "bpftool build" test doesn't depend on the kernel build
at all, it can be implemented as a new workflow independent of
everything. You don't even need new code in libbpf/ci for that, just a
new .yaml in kernel-patches/vmtest/.github/workflows/, and maybe a new
shell script.  And it can probably run on github-provided ubuntu
runners.  That would be a right way to implement it, given the current
state of things IMO.

> 
>> I'm thinking the whole "kernel build" workflow should be refactored
>> into either separate selftests build jobs, or some post processing
>> that looks at what artifacts where built successfully and reports
>> failures. This will provide a little more friendly UI/UX. You're
>> welcome to look into that if you're interested. But that's not
>> directly relevant to the changes you're proposing.
> 
> Trying to rephrase it to make sure I get your point, the goal would be
> to have green/red checks specifically on selftests builds, rather than a
> global green/red check on the macro job "build kernel and selftests" ?
> Or does it go further than that ?
> 
> If I get it correctly, yes I'd be glad to try and help on that, as a
> separate task.

Ok, let me expand on this. Thinking out loud below.

As you correctly noted, currently we have a kernel build + selftests
build clumped together in a single job. I don't know if this was
intentional design when it was set up, but it's what's there.

The advantage of "build many things in one job" compared to "one job -
one artifact" extreme is in the cost of transition between the jobs on
the performance side (artifacts upload/download, potentially new
runner cold start etc.), and in the complexity of the independent jobs
(you have to set up prereqs every time, for example).

The disadvantage is in the coupling of the dependencies. For example,
currently if selftests/bpf build fails, veristat jobs don't run,
because they depend on the combined build job. But they actually *can*
run if the kernel build succeeded, but selftests build failed.

An argument can be made that if your CI run is red, you're going to
dig into the logs anyways, so this is all bikeshedding. But I think a
clear "kernel build failed" or "bpftool build failed" signal saves a
bit of time for everyone.

An idea I have on how we could have the cake and eat it too - avoid
the cost while keeping the clear signal - is to keep independent build
steps in the combined job as allowed to fail, so the job always
succeeds. And then have a separate check for "this artifact is missing
- that means the build failed - here is the log", either in a
dependent job or as a lightweight "check" job.

It's not free though: some cost will remain in the complexity of the
combined job, and the checks.

Validating this idea requires github .yaml hackery (not a fun activity
in general) and experimentation. It looks doable to me, but there
might be problems with it I haven't thought about.

> 
> Alexis
> 
>> [1] https://lore.kernel.org/bpf/20260804170156.1709916-1-nickolay.lysenko@gmail.com/
>> [2] https://github.com/kernel-patches/vmtest/pull/502#discussion_r3724931742
>>
>>
>>>
>>> Alexis
>>>
>>> [0] https://github.com/kernel-patches/vmtest/pull/517
>>> [1] https://github.com/libbpf/ci/pull/236
>>> [2] https://github.com/kernel-patches/bpf/pull/13368
>>>
> 
> 
> 
>