Re: [iptables PATCH 2/4] tests: add `NOMATCH` test result

[Phil Sutter <phil@nwl.cc>]

Hi Jeremy,

On Sun, Feb 20, 2022 at 01:10:29PM +0000, Jeremy Sowden wrote:
> On 2022-02-14, at 11:01:13 +0100, Phil Sutter wrote:
> > On Sat, Feb 12, 2022 at 04:58:30PM +0000, Jeremy Sowden wrote:
> > > Currently, there are two supported test results: `OK` and `FAIL`.
> > > It is expected that either the iptables command fails, or it
> > > succeeds and dumping the rule has the correct output.  However, it
> > > is possible that the command may succeed but the output may not be
> > > correct.  Add a `NOMATCH` result to cover this outcome.
> >
> > Hmm. Wouldn't it make sense to extend the scope of LEGACY/NFT keywords
> > to output checks as well instead of introducing a new one? I think we
> > could cover expected output that way by duplicating the test case with
> > different expected output instead of marking it as unspecific "may
> > produce garbage".
> 
> Something like the following?  One reason why I went with the `NOMATCH`
> result is that in the two divergent test-cases, there is no -nft output
> to match.  We can make that work by just using the empty string as the
> alternative output since that will match anything.  I don't think it's
> ideal, but it's simpler than overhauling the matching code for what is a
> rare corner case.

Thanks for compiling the patch. What I had in mind was to merge result
checks of failing rule with output mismatch, but I realize this would
likely turn into a mess.

[...]
> In the case of tests which have no output to match, we leave the last
> field empty:
> 
>   -j EXAMPLE-TARGET --example-option;=;OK;LEGACY;

A non-empty rule leading to empty output is a bug, IMHO.

[...]
> --- a/extensions/libxt_NFLOG.t
> +++ b/extensions/libxt_NFLOG.t
> @@ -5,8 +5,8 @@
>  -j NFLOG --nflog-group 0;-j NFLOG;OK
>  # `--nflog-range` is broken and only supported by xtables-legacy.
>  # It has been superseded by `--nflog--group`.
> --j NFLOG --nflog-range 1;=;OK;LEGACY;NOMATCH
> --j NFLOG --nflog-range 4294967295;=;OK;LEGACY;NOMATCH
> +-j NFLOG --nflog-range 1;=;OK;LEGACY;
> +-j NFLOG --nflog-range 4294967295;=;OK;LEGACY;

The crucial detail here is that an expected output of "-j NFLOG" is
trivial and str::find() won't complain about extra output.

Given that we're discussing corner cases and what I had in mind has its
own downsides, I guess the status quo is fine at least for now. Sorry
for the fuss!

Cheers, Phil