Re: [PATCH] selftests: Skip BPF seftests by default

[Shuah Khan <skhan@linuxfoundation.org>]

On 12/17/20 6:07 AM, Mark Brown wrote:
> On Wed, Dec 16, 2020 at 04:05:58PM -0600, Seth Forshee wrote:
>> On Thu, Dec 10, 2020 at 06:52:33PM +0000, Mark Brown wrote:
> 
>>> as part of the wider kselftest build by specifying SKIP_TARGETS,
>>> including setting an empty SKIP_TARGETS to build everything.  They can
>>> also continue to build the BPF selftests individually in cases where
>>> they are specifically focused on BPF.
> 
>> Why not just remove the line which adds bpf to TARGETS? This has the
>> same effect, but doesn't require an emtpy SKIP_TARGETS to run them. We
>> have testing scripts which use 'make TARGETS=bpf ...' which will have to
>> be updated, and I doubt we are the only ones.
> 

I would prefer leaving bpf in the main Makefile TARGETS. This will be
useful to users that have their systems setup for bpf builds.

>> I also feel like this creates confusing semantics around SKIP_TARGETS.
>> If I don't supply a value then I don't get the bpf selftests, but then
>> if I try to use SKIP_TARGETS to skip some other test suddenly I do get
>> them. That's counterintuitive.
> 
> That's what I did first, it's also messy just differently.  If you
> don't add bpf to TARGETS then if you do what's needed to get it building
> it becomes inconvenient to run it as part of running everything else at
> the top level since you need to enumerate all the targets.  It felt like
> skipping is what we're actually doing here and it seems like those
> actively working with BPF will be used to having to update things in
> their environment.  People who start using SKIP_TARGETS are *probably*
> going to find out about it from the Makefile anyway so will see the
> default that's there.
> 
> Fundamentally having such demanding build dependencies is always going
> to result in some kind of mess, it's just where we push it.
> 
>> I also wanted to point out that the net/test_bpf.sh selftest requires
>> having the test_bpf module from the bpf selftest build. So when the bpf
>> selftests aren't built this test is guaranteed to fail. Though it would
>> be nice if the net selftests didn't require building the bpf self tests
>> in order to pass.
> 
> Right, that's a separate issue - the net tests should really skip that
> if they don't have BPF, as we do for other runtime detectable
> dependencies.  It's nowhere near as severe as failing to build though.
> 

Correct. This has to be handled as a run-time dependency check and skip
instead of fail.

thanks,
-- Shuah