Re: [PATCH v4 00/10] selftests/bpf: Tolerate partial builds across kernel configs

["Ricardo B. Marlière" <rbm@suse.com>]

Hi Alan,

On Fri Apr 10, 2026 at 7:06 AM -03, Alan Maguire wrote:
> On 06/04/2026 19:22, Ricardo B. Marlière wrote:
>> Currently the BPF selftests can only be built by using the minimum kernel
>> configuration defined in tools/testing/selftests/bpf/config*. This poses a
>> problem in distribution kernels that may have some of the flags disabled or
>> set as module. For example, we have been running the tests regularly in
>> openSUSE Tumbleweed [1] [2] but to work around this fact we created a
>> special package [3] that build the tests against an auxiliary vmlinux with
>> the BPF Kconfig. We keep a list of known issues that may happen due to,
>> amongst other things, configuration mismatches [4] [5].
>> 
>> The maintenance of this package is far from ideal, especially for
>> enterprise kernels. The goal of this series is to enable the common usecase
>> of running the following in any system:
>> 
>> ```sh
>> make -C tools/testing/selftests install \
>>         SKIP_TARGETS= \
>>         TARGETS=bpf \
>>         O=/lib/modules/$(uname -r)/build
>> ```
>> 
>> As an example, the following script targeting a minimal config can be used
>> for testing:
>> 
>> ```sh
>> make defconfig
>> scripts/config --file .config \
>>                --enable DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
>>                --enable DEBUG_INFO_BTF \
>>                --enable BPF_SYSCALL \
>>                --enable BPF_JIT
>> make olddefconfig
>> make -j$(nproc)
>> make -j$(nproc) -C tools/testing/selftests SKIP_TARGETS= TARGETS=bpf install
>> ```
>> 
>> This produces a test_progs binary with 579 subtests, against the total of
>> 708. Many of them will still fail or be skipped at runtime due to lack of
>> symbols, but at least there will be a clear way of building the tests.
>>
>
> hi Ricardo, I've been looking at this series and one concern I had was if
> tolerating failures to build with incomplete config would create problems
> in detecting failures when the config is complete, i.e. in BPF CI. So for
> example if say a BPF skeleton didn't compile when it really should due to 
> an introduced bug, we wouldn't want to drive on in all cases.

Agreed.

>
> However, from what I can see, CI held up well with these changes in place [1].
>
> That said it might be nice to preserve a "strict" mode which fails when
> we hit compilation issues etc, failing as we do today; I couldn't find any existing 
> general kselftest concept that fits with that but it would be valuable I think
> if possible. Perhaps we could default to non-strict mode when a make -C is initiated
> or something like that, and continue to operate strictly when building directly
> in tools/testing/selftests/bpf?
>
> I ran the changes through the usual workflow I use - build bpf selftests at
> tools/testing/selftests/bpf level and run ./test_progs ; no regressions were
> observed, so feel free to add for the series
>
> Tested-by: Alan Maguire <alan.maguire@oracle.com>

Thanks for testing!

>
> There are some comments on sashiko.dev [2] that would be worth looking through
> too, in particular the concern about O= and M= mentioned for patch 1.

I will send v5 with those addressed and possibly a strict-mode switch.

>
> [1] https://github.com/kernel-patches/bpf/actions/runs/24168181680
> [2] https://sashiko.dev/#/patchset/20260406-selftests-bpf_misconfig-v4-0-9914f50efdf7%40suse.com
>  > [1]: https://openqa.opensuse.org/tests/5811715
>> [2]: https://openqa.opensuse.org/tests/5811730
>> [3]: https://src.opensuse.org/rmarliere/kselftests
>> [4]: https://github.com/openSUSE/kernel-qe/blob/main/kselftests_known_issues.yaml
>> [5]: https://openqa.opensuse.org/tests/5811730/logfile?filename=run_kselftests-config_mismatches.txt
>> 
>> Assisted-by: {codex,claude}
>> Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
>> ---
>> Changes in v4:
>> - Drop the test_kmods kselftest module flow patch: lib.mk gen_mods_dir
>>   invokes $(MAKE) -C $(TEST_GEN_MODS_DIR) without forwarding
>>   RESOLVE_BTFIDS, breaking ASAN and GCC BPF CI builds (Makefile.modfinal
>>   cannot find resolve_btfids in the kbuild output tree)
>> - Link to v3:
>>   https://patch.msgid.link/20260406-selftests-bpf_misconfig-v3-0-587a1114263c@suse.com
>> 
>> Changes in v3:
>> - Split test_kmods patch into two: fix KDIR handling (O= passthrough,
>>   EXTRA_CFLAGS/EXTRA_LDFLAGS clearing) and wire into lib.mk via
>>   TEST_GEN_MODS_DIR
>> - Pass O= through to the kernel module build so artifacts land in the
>>   output tree, not the source tree
>> - Clear EXTRA_CFLAGS and EXTRA_LDFLAGS when invoking the kernel build to
>>   prevent host flags (e.g. -static) leaking into module compilation
>> - Replace the bespoke test_kmods pattern rule with lib.mk module
>>   infrastructure (TEST_GEN_MODS_DIR); lib.mk now drives build and clean
>>   lifecycle
>> - Make the .ko copy step resilient: emit SKIP instead of failing when a
>>   module is absent
>> - Expand the uprobe weak stub comment in bpf_cookie.c to explain why
>>   noinline is required
>> - Link to v2:
>>   https://patch.msgid.link/20260403-selftests-bpf_misconfig-v2-0-f06700380a9d@suse.com
>> 
>> Changes in v2:
>> - Skip test_kmods build/clean when KDIR directory does not exist
>> - Use `Module.symvers` instead of `.config` for in-tree detection
>> - Fix skeleton order-only prereqs commit message
>> - Guard BTFIDS step when .test.o is absent
>> - Add `__weak stack_mprotect()` stubs in `bpf_cookie.c` and `iters.c`
>> - Link to v1:
>>   https://patch.msgid.link/20260401-selftests-bpf_misconfig-v1-0-3ae42c0af76f@suse.com
>> 
>> ---
>> Ricardo B. Marlière (10):
>>       selftests/bpf: Fix test_kmods KDIR to honor O= and distro kernels
>>       selftests/bpf: Tolerate BPF and skeleton generation failures
>>       selftests/bpf: Avoid rebuilds when running emit_tests
>>       selftests/bpf: Make skeleton headers order-only prerequisites of .test.d
>>       selftests/bpf: Tolerate test file compilation failures
>>       selftests/bpf: Allow test_progs to link with a partial object set
>>       selftests/bpf: Tolerate benchmark build failures
>>       selftests/bpf: Provide weak definitions for cross-test functions
>>       selftests/bpf: Skip tests whose objects were not built
>>       selftests/bpf: Tolerate missing files during install
>> 
>>  tools/testing/selftests/bpf/Makefile               | 113 ++++++++++++++-------
>>  .../testing/selftests/bpf/prog_tests/bpf_cookie.c  |  24 ++++-
>>  tools/testing/selftests/bpf/prog_tests/iters.c     |  10 +-
>>  tools/testing/selftests/bpf/test_kmods/Makefile    |  15 ++-
>>  tools/testing/selftests/bpf/test_progs.c           |   7 +-
>>  5 files changed, 120 insertions(+), 49 deletions(-)
>> ---
>> base-commit: a1aa9ef47c299c5bbc30594d3c2f0589edf908e6
>> change-id: 20260401-selftests-bpf_misconfig-4c33ef5c56da
>> 
>> Best regards,
>> --  
>> Ricardo B. Marlière <rbm@suse.com>
>> 
>>