[PATCH nft 0/2] tests/shell: add mechanism for skipping (for no JSON and slow)

[Thomas Haller <thaller@redhat.com>]

Add a mechanism to skip tests. Two cases are implemented:

- skip thest that require JSON support, if nft binary is build without.

- the user can choose to skip long-running tests via the (-Q|--quick) option.
  Most tests are fast, so by skipping 8 of 373 tests, I can run the
  test suite in 7 seconds.

This is inspired and related to Florian's "feature probing" patchset.
And done with the intent, that those patches could integrate in a
common mechanism. Florian's patches are NOT obsoleted by this
(although, they would require rebase/adjustment).

Changes/notes compared to Florian's approach:

- Florian's patchset mostly called the variable NFT_HAVE_xxx (although,
  the commit message also called it NFT_TEST_HAVE_xxx). I choose
  NFT_TEST_HAVE_* name to make it clear that the environment variables are
  related to the tests.

- The evalution of NFT_TEST_HAVE_json=y|n is hard coded in "run-test.sh"
  because it seems special enough. But we certainly can combine that with
  the generic mechanism to load features from the "features/" directory.

- NFT_TEST_HAVE_xxx can also be set by the user. That overrides the
  feature detection. The point is to force the detection for manual
  testing.

- let "test-wrapper.sh" can parse tags like

    # NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)
    # NFT_TEST_SKIP(NFT_TEST_SKIP_slow)

  to simplify marking tests with the skip/requires option.

Thomas Haller (2):
  tests/shell: skip tests if nft does not support JSON mode
  tests/shell: add "--quick" option to skip slow tests (via
    NFT_TEST_SKIP_slow=y)

 tests/shell/helpers/test-wrapper.sh           | 51 ++++++++++++++++-
 tests/shell/run-tests.sh                      | 56 ++++++++++++++++++-
 .../shell/testcases/json/0001set_statements_0 |  2 +
 tests/shell/testcases/json/0002table_map_0    |  2 +
 .../testcases/json/0003json_schema_version_0  |  2 +
 .../testcases/json/0004json_schema_version_1  |  2 +
 .../shell/testcases/json/0005secmark_objref_0 |  2 +
 tests/shell/testcases/json/0006obj_comment_0  |  2 +
 tests/shell/testcases/json/netdev             |  9 ++-
 .../listing/0021ruleset_json_terse_0          | 13 ++++-
 .../maps/0004interval_map_create_once_0       |  8 +++
 .../testcases/maps/0018map_leak_timeout_0     |  2 +
 tests/shell/testcases/maps/vmap_timeout       |  2 +
 .../testcases/sets/0043concatenated_ranges_0  |  2 +
 .../testcases/sets/0044interval_overlap_0     |  2 +
 .../testcases/sets/0044interval_overlap_1     |  2 +
 tests/shell/testcases/sets/automerge_0        |  2 +
 tests/shell/testcases/transactions/0049huge_0 | 14 ++++-
 tests/shell/testcases/transactions/30s-stress |  2 +
 19 files changed, 168 insertions(+), 9 deletions(-)

-- 
2.41.0