[PATCH nft 1/2] tests/shell: exit 77 from "run-tests.sh" if all tests were skipped

[PATCH nft 1/2] tests/shell: exit 77 from "run-tests.sh" if all tests were skipped

[Thomas Haller]

If there are multiple tests and some of them pass and some are skipped,
the overall result should be success (zero). Because likely the user
just selected a bunch of tests (or all of them). So skipping some tests
does not mean that the entire run is not a success.

However, if all tests are skipped, then mark the overall result as
skipped too. The more common case is if you only run one single test,
then we want to know, that the test didn't run.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index bdca0ee1fa0b..188ac89ca1de 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -728,4 +728,10 @@ if [ "$failed" -gt 0 -o "$NFT_TEST_KEEP_LOGS" = y ] ; then
 	NFT_TEST_TMPDIR=
 fi
 
-[ "$failed" -eq 0 ]
+if [ "$failed" -gt 0 ] ; then
+	exit 1
+elif [ "$ok" -eq 0 -a "$skipped" -gt 0 ] ; then
+	exit 77
+else
+	exit 0
+fi
-- 
2.41.0

[PATCH nft 2/2] tests/shell: accept $NFT_TEST_TMPDIR_TAG for the result directory

[Thomas Haller]

We allow the user to set "$TMPDIR" to affect where the "nft-test.*"
directory is created. However, we don't allow the user to specify the
exact location, so the user doesn't really know which directory was
created.

One remedy is that the test will also create the symlink
"$TMPDIR/nft-test.latest.$USER" to point to the last test result.
However, if you run multiple tests in parallel, that is not reliable to
find the test results.

Accept $NFT_TEST_TMPDIR_TAG and use it as part of the generated
filename. That way, the caller can set it to a unique tag, and find the
directory later based on that. For example

  export TMPDIR=/tmp
  export NFT_TEST_TMPDIR_TAG=".$(uuidgen)"
  ./tests/shell/run-tests.sh
  ls -lad "$TMPDIR/nft-test."*"$NFT_TEST_TMPDIR_TAG"*/

will work reliably -- as long as the tag is chosen uniquely.

The reason to not allow the user to specify the directory name directly,
is because we want that tests results follow the well-known pattern
"/tmp/nft-test*".

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/run-tests.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 188ac89ca1de..f75505f7f1b4 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -433,7 +433,7 @@ cleanup_on_exit() {
 }
 trap cleanup_on_exit EXIT
 
-NFT_TEST_TMPDIR="$(mktemp --tmpdir="$_TMPDIR" -d "nft-test.$(date '+%Y%m%d-%H%M%S.%3N').XXXXXX")" ||
+NFT_TEST_TMPDIR="$(mktemp --tmpdir="$_TMPDIR" -d "nft-test.$(date '+%Y%m%d-%H%M%S.%3N')$NFT_TEST_TMPDIR_TAG.XXXXXX")" ||
 	msg_error "Failure to create temp directory in \"$_TMPDIR\""
 chmod 755 "$NFT_TEST_TMPDIR"
 
-- 
2.41.0