Re: [PATCH bpf-next 1/2] selftests/bpf: skip verifier tests that depend on CONFIG_CGROUP_BPF

[Stanislav Fomichev <sdf@fomichev.me>]

On 12/12, Alexei Starovoitov wrote:
> On Wed, Dec 12, 2018 at 09:23:30PM +0000, Edward Cree wrote:
> > On 12/12/18 20:13, Alexei Starovoitov wrote:
> > > that won't work either.
> > > "bpf feature set" is a lot more than number of program
> > > and map types the kernel supports. There are all sorts of
> > > helper combinations, hooks, and verifier improvements.
> > > test_verifier.c must test all that.
> > > I don't think there is a way to make usptream test_verfier.c
> > > not to report failure on older kernels.
> > But it's not just older kernels; AIUI there are config options
> >  that also affect this.  Are you saying that test_verifier
> >  should only be expected to run / pass on allyesconfig kernels?
> 
> not allyesconfig, but allbpfyesconfig.
Yes, to reiterate, this patch series was mostly about running verifier
against the kernel where we don't have all bpf features enabled (to make
sure we also don't break them by having some (un)related inhouse patches).

> > I think that for the cases where we _can_ do it easily (which
> >  seems to be precisely things like prog_type which don't require
> >  any additional annotation of test cases) we should skip tests
> >  that aren't supported by the running kernel.
> 
> Submitters of bpf patches need to enable all bpf features in
> their kernels and make sure that all of the selftests/bpf
> are still passing.
> "but I don't use XDP today" or "LLVM is hard to install"
> is not an excuse.
> 
> Before I apply any patch I build and test manually.
> If anything breaks means folks didn't do enough testing
> and karma suffers.
Your use case would still work, you just make sure there are no skips.
Karma would still suffer if you run the tests and they fail, but for the
submitter they were skipped ;-)

To summarize, I like your idea about doing runtime tests and I think I
can make it work quite nicely without any config_disabled ugliness by
looking at the prog_type of each test.
I can send an RFC patch series out if there still a small chance you could
take it, but if you've already set you mind, I'd just keep them
internally. So let me know if you have a hard NACK on the runtime probing
approach or there is still some wiggle room.