Re: [nft PATCH 6/6] Makefile: Enable support for 'make check'

[Pablo Neira Ayuso <pablo@netfilter.org>]

On Thu, Aug 07, 2025 at 01:51:09PM +0200, Phil Sutter wrote:
> On Wed, Aug 06, 2025 at 07:05:02PM +0200, Pablo Neira Ayuso wrote:
> > On Fri, Aug 01, 2025 at 06:11:05PM +0200, Phil Sutter wrote:
> > > Add the various testsuite runners to TESTS variable and have make call
> > > them with RUN_FULL_TESTSUITE=1 env var.
> > > 
> > > Signed-off-by: Phil Sutter <phil@nwl.cc>
> > > ---
> > >  Makefile.am | 6 ++++++
> > >  1 file changed, 6 insertions(+)
> > > 
> > > diff --git a/Makefile.am b/Makefile.am
> > > index ba09e7f0953d5..4fb75b85a5d59 100644
> > > --- a/Makefile.am
> > > +++ b/Makefile.am
> > > @@ -409,5 +409,11 @@ EXTRA_DIST += \
> > >  	tests \
> > >  	$(NULL)
> > >  
> > > +AM_TESTS_ENVIRONMENT = RUN_FULL_TESTSUITE=1; export RUN_FULL_TESTSUITE;
> > 
> > I use make distcheck to build the tarballs.
> > 
> > I would prefer not to run the tests at the time of the release
> > process, I always do this before release, but I prefer not to inline
> > this to the release process.
> 
> Oh, good to know. Running just 'make dist' is no option for you?

I can just modify the script to do so, no idea on the implications.

Or wait for one more hour for the test run to finish during the
release process.

> BTW: There is the same situation with iptables, though if called as
> unprivileged user there is only the xlate test suite which runs (and
> quickly finishes).
> 
> > Maybe we can make this work this way?
> > 
> >   export RUN_FULL_TESTSUITE=1; make check
> > 
> > so make check is no-op without this variable?
> > 
> > Does this make sense to you?
> 
> It seems odd to enable 'make check' only to disable it again, but
> there's still added value in it.
> 
> I'm currently looking into distcheck-hook and DISTCHECK_CONFIGURE_FLAGS
> in order to identify the caller to 'make check' call.
> 
> An alternative would be to drop fake root functionality from shell
> test suite, then it would skip just like all the other test suites if run
> as non-root (assuming you don't run 'make distcheck' as root).

Looking at the current release script I have, it all runs as non-root.

Maybe simply as a run-all.sh under nftables/tests/?

It is not so elegant as make check, but still allows for running _all_
tests with minimal changes.

tests/build is slow but very useful.

> Another side-quest extracted from this mail: There's an odd failure from
> shell test suite when run by 'make distcheck':
> 
> | E: cannot execute nft command: ../../tests/shell/../../src/nft
> | ERROR tests/shell/run-tests.sh (exit status: 99)
> 
> > > +TESTS = tests/json_echo/run-test.py \
> > > +	tests/monitor/run-tests.sh \
> > > +	tests/py/nft-test.py \
> > > +	tests/shell/run-tests.sh
> > 
> > BTW, there are also tests/build/ that are slow but useful, that helped
> > me find this:
> > 
> > https://git.netfilter.org/nftables/commit/?id=0584f1c1c2073ff082badc7b49ed667de41002d9
> 
> Ah, I just "discovered" that bug as well, it surfaces from the build
> performed by 'make distcheck', too.
> 
> Thanks, Phil