Re: [PATCH nft v3 2/6] tests/shell: check and generate JSON dump files

[Thomas Haller <thaller@redhat.com>]

On Wed, 2023-11-15 at 10:54 +0100, Pablo Neira Ayuso wrote:
> On Wed, Nov 15, 2023 at 09:24:27AM +0100, Florian Westphal wrote:
> > Thomas Haller <thaller@redhat.com> wrote:
> > > The rules after a successful test are good opportunity to test
> > > `nft -j list ruleset` and `nft -j --check`. This quite possibly
> > > touches
> > > code paths that are not hit by other tests yet.
> > 
> > This series looks good to me, I'll apply it in the next few hours
> > if
> > noone else takes any action by then.
> 
> Just a question, patch 3 is missing in patchwork. I guess it is too
> big.


Yes, it's not on the list, as it's too large. I CC-ed you and Florian
on patch 2/6.

You can also find it here:
https://gitlab.freedesktop.org/thaller/nftables/-/commit/b0edc64d005510b8c3db8a8ebe496a8296271bf4.patch

> 
> My understanding is that this performs the json tests if nft comes
> with
> json support.

right.

> 
> I wanted to give this a run, description says a few tests are
> failing.

For tests that fail, the patch 2/6 does not add a `.json-nft` file.

If you get any failure, that's wrong. Then the corresponding .json-nft
should be excluded from patch 2/6.

> Last time we talked it is chain binding support, then there is a good
> number of tests that are going to fail (or there is a mechanism to
> temporarily disable json tests for this without losing coverage?).

If chain binding support is missing, then that is detected via the
common mechanism (NFT_TEST_HAVE_chain_binding=n) and the test will be
marked as SKIPPED.

SKIPPED tests don't get their .nft dump checked. The same for .json-nft
files.

(of course, it also honors NFT_TEST_HAVE_json=n to skip the check).

> 
> What is the current output from tests? I wanted to make this run
> myself so I don't need to ask.

it's the same as with .nft files.

If the .nft/.json-nft dump does not match, the test fails with [DUMP
FAIL]. As always, you can find the result data in /tmp.

I find it most useful to run

  grep --color=always ^ -a -R /tmp/nft-test.latest.*/ | less -R

but whatever works for you.


Also, the test wrapper will call `nft --check -f x.nft` and `nft -j --
check -f x.json-nft` after tests. When those fail, you'll see [CHK
DUMP].

You can see that with:

  DUMPGEN=all ./tests/shell/run-tests.sh tests/shell/testcases/chains/0011endless_jump_loop_1



Thomas