[PATCH nft 1/1] tests: prettify JSON in test output and add helper

[PATCH nft 1/1] tests: prettify JSON in test output and add helper

[Thomas Haller]

- add helper script "json-pretty.sh" for prettify/format JSON.
  It uses either `jq` or a `python` fallback. In my tests, they
  produce the same output, but the output is not guaranteed to be
  stable. This is mainly for informational purpose.

- add a "json-diff-pretty.sh" which prettifies two JSON inputs and
  shows a diff of them.

- in "test-wrapper.sh", after the check for a .json-nft dump fails, also
  call "json-diff-pretty.sh" and write the output to "ruleset-diff.json.pretty".
  This is beside "ruleset-diff.json", which contains the original diff.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/json-diff-pretty.sh | 17 +++++++++++++++++
 tests/shell/helpers/json-pretty.sh      | 17 +++++++++++++++++
 tests/shell/helpers/test-wrapper.sh     |  4 ++++
 3 files changed, 38 insertions(+)
 create mode 100755 tests/shell/helpers/json-diff-pretty.sh
 create mode 100755 tests/shell/helpers/json-pretty.sh

diff --git a/tests/shell/helpers/json-diff-pretty.sh b/tests/shell/helpers/json-diff-pretty.sh
new file mode 100755
index 000000000000..bebb7e8ed006
--- /dev/null
+++ b/tests/shell/helpers/json-diff-pretty.sh
@@ -0,0 +1,17 @@
+#!/bin/bash -e
+
+BASEDIR="$(dirname "$0")"
+
+[ $# -eq 2 ] || (echo "$0: expects two JSON files as arguments" ; exit 1)
+
+FILE1="$1"
+FILE2="$2"
+
+pretty()
+{
+	"$BASEDIR/json-pretty.sh" < "$1" 2>&1 || :
+}
+
+echo "Cmd: \"$0\" \"$FILE1\" \"$FILE2\""
+diff -u "$FILE1" "$FILE2" 2>&1 || :
+diff -u <(pretty "$FILE1") <(pretty "$FILE2") 2>&1 || :
diff --git a/tests/shell/helpers/json-pretty.sh b/tests/shell/helpers/json-pretty.sh
new file mode 100755
index 000000000000..0d6972b81e2f
--- /dev/null
+++ b/tests/shell/helpers/json-pretty.sh
@@ -0,0 +1,17 @@
+#!/bin/bash -e
+
+# WARNING: the output is not guaranteed to be stable.
+
+if command -v jq &>/dev/null ; then
+	# If we have, use `jq`
+	exec jq
+fi
+
+# Fallback to python.
+exec python -c '
+import json
+import sys
+
+parsed = json.load(sys.stdin)
+print(json.dumps(parsed, indent=2))
+'
diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
index 9e8e60581890..4ffc48184dd7 100755
--- a/tests/shell/helpers/test-wrapper.sh
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -202,6 +202,10 @@ if [ "$rc_test" -ne 77 -a "$dump_written" != y ] ; then
 	fi
 	if [ "$NFT_TEST_HAVE_json" != n -a -f "$JDUMPFILE" ] ; then
 		if ! $DIFF -u "$JDUMPFILE" "$NFT_TEST_TESTTMPDIR/ruleset-after.json" &> "$NFT_TEST_TESTTMPDIR/ruleset-diff.json" ; then
+			"$NFT_TEST_BASEDIR/helpers/json-diff-pretty.sh" \
+				"$JDUMPFILE" \
+				"$NFT_TEST_TESTTMPDIR/ruleset-after.json" \
+				2>&1 > "$NFT_TEST_TESTTMPDIR/ruleset-diff.json.pretty"
 			show_file "$NFT_TEST_TESTTMPDIR/ruleset-diff.json" "Failed \`$DIFF -u \"$JDUMPFILE\" \"$NFT_TEST_TESTTMPDIR/ruleset-after.json\"\`" >> "$NFT_TEST_TESTTMPDIR/rc-failed-dump"
 			rc_dump=1
 		else
-- 
2.42.0

Re: [PATCH nft 1/1] tests: prettify JSON in test output and add helper

[Pablo Neira Ayuso]

Hi,

On Wed, Nov 22, 2023 at 12:19:40PM +0100, Thomas Haller wrote:
> - add helper script "json-pretty.sh" for prettify/format JSON.
>   It uses either `jq` or a `python` fallback. In my tests, they
>   produce the same output, but the output is not guaranteed to be
>   stable. This is mainly for informational purpose.
> 
> - add a "json-diff-pretty.sh" which prettifies two JSON inputs and
>   shows a diff of them.
> 
> - in "test-wrapper.sh", after the check for a .json-nft dump fails, also
>   call "json-diff-pretty.sh" and write the output to "ruleset-diff.json.pretty".
>   This is beside "ruleset-diff.json", which contains the original diff.

One silly question: Does the prettify hightlights the difference?

tests/py clearly shows what is the difference in the JSON diff that
quickly helps you identify what is missing.

Thanks!

> Signed-off-by: Thomas Haller <thaller@redhat.com>
> ---
>  tests/shell/helpers/json-diff-pretty.sh | 17 +++++++++++++++++
>  tests/shell/helpers/json-pretty.sh      | 17 +++++++++++++++++
>  tests/shell/helpers/test-wrapper.sh     |  4 ++++
>  3 files changed, 38 insertions(+)
>  create mode 100755 tests/shell/helpers/json-diff-pretty.sh
>  create mode 100755 tests/shell/helpers/json-pretty.sh
> 
> diff --git a/tests/shell/helpers/json-diff-pretty.sh b/tests/shell/helpers/json-diff-pretty.sh
> new file mode 100755
> index 000000000000..bebb7e8ed006
> --- /dev/null
> +++ b/tests/shell/helpers/json-diff-pretty.sh
> @@ -0,0 +1,17 @@
> +#!/bin/bash -e
> +
> +BASEDIR="$(dirname "$0")"
> +
> +[ $# -eq 2 ] || (echo "$0: expects two JSON files as arguments" ; exit 1)
> +
> +FILE1="$1"
> +FILE2="$2"
> +
> +pretty()
> +{
> +	"$BASEDIR/json-pretty.sh" < "$1" 2>&1 || :
> +}
> +
> +echo "Cmd: \"$0\" \"$FILE1\" \"$FILE2\""
> +diff -u "$FILE1" "$FILE2" 2>&1 || :
> +diff -u <(pretty "$FILE1") <(pretty "$FILE2") 2>&1 || :
> diff --git a/tests/shell/helpers/json-pretty.sh b/tests/shell/helpers/json-pretty.sh
> new file mode 100755
> index 000000000000..0d6972b81e2f
> --- /dev/null
> +++ b/tests/shell/helpers/json-pretty.sh
> @@ -0,0 +1,17 @@
> +#!/bin/bash -e
> +
> +# WARNING: the output is not guaranteed to be stable.
> +
> +if command -v jq &>/dev/null ; then
> +	# If we have, use `jq`
> +	exec jq
> +fi
> +
> +# Fallback to python.
> +exec python -c '
> +import json
> +import sys
> +
> +parsed = json.load(sys.stdin)
> +print(json.dumps(parsed, indent=2))
> +'
> diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
> index 9e8e60581890..4ffc48184dd7 100755
> --- a/tests/shell/helpers/test-wrapper.sh
> +++ b/tests/shell/helpers/test-wrapper.sh
> @@ -202,6 +202,10 @@ if [ "$rc_test" -ne 77 -a "$dump_written" != y ] ; then
>  	fi
>  	if [ "$NFT_TEST_HAVE_json" != n -a -f "$JDUMPFILE" ] ; then
>  		if ! $DIFF -u "$JDUMPFILE" "$NFT_TEST_TESTTMPDIR/ruleset-after.json" &> "$NFT_TEST_TESTTMPDIR/ruleset-diff.json" ; then
> +			"$NFT_TEST_BASEDIR/helpers/json-diff-pretty.sh" \
> +				"$JDUMPFILE" \
> +				"$NFT_TEST_TESTTMPDIR/ruleset-after.json" \
> +				2>&1 > "$NFT_TEST_TESTTMPDIR/ruleset-diff.json.pretty"
>  			show_file "$NFT_TEST_TESTTMPDIR/ruleset-diff.json" "Failed \`$DIFF -u \"$JDUMPFILE\" \"$NFT_TEST_TESTTMPDIR/ruleset-after.json\"\`" >> "$NFT_TEST_TESTTMPDIR/rc-failed-dump"
>  			rc_dump=1
>  		else
> -- 
> 2.42.0
> 

Re: [PATCH nft 1/1] tests: prettify JSON in test output and add helper

[Thomas Haller]

On Wed, 2023-11-22 at 13:33 +0100, Pablo Neira Ayuso wrote:
> Hi,
> 
> On Wed, Nov 22, 2023 at 12:19:40PM +0100, Thomas Haller wrote:
> > - add helper script "json-pretty.sh" for prettify/format JSON.
> >   It uses either `jq` or a `python` fallback. In my tests, they
> >   produce the same output, but the output is not guaranteed to be
> >   stable. This is mainly for informational purpose.
> > 
> > - add a "json-diff-pretty.sh" which prettifies two JSON inputs and
> >   shows a diff of them.
> > 
> > - in "test-wrapper.sh", after the check for a .json-nft dump fails,
> > also
> >   call "json-diff-pretty.sh" and write the output to "ruleset-
> > diff.json.pretty".
> >   This is beside "ruleset-diff.json", which contains the original
> > diff.
> 
> One silly question: Does the prettify hightlights the difference?

Yes. That that is the idea.

> 
> tests/py clearly shows what is the difference in the JSON diff that
> quickly helps you identify what is missing.


As always, you will find some result files in /tmp/nft-
test.latest.$USER/, which I usually read with

  $ grep --color=always ^ -aR /tmp/nft-test.latest.*/ | less -R

there will be a new file there, named "ruleset-diff.json.pretty".

It contains the content of "./tests/shell/helpers/json-diff-pretty.sh"
output.


For example:

  $ cp tests/shell/testcases/bitwise/dumps/0040mark_binop_2.json-nft tests/shell/testcases/bitwise/dumps/0040mark_binop_3.json-nft
  $ ./tests/shell/run-tests.sh tests/shell/testcases/bitwise/0040mark_binop_3


leaves a file

     /tmp/nft-test.latest.*/test-tests-shell-testcases-bitwise-0040mark_binop_3.1/ruleset-diff.json.pretty


with the following content:


Cmd: "./tests/shell/helpers/json-diff-pretty.sh" "tests/shell/testcases/bitwise/dumps/0040mark_binop_3.json-nft" "/tmp/nft-test.20231122-135045.550.28uKHu/test-tests-shell-testcases-bitwise-0040mark_binop_3.1/ruleset-after.json"
--- tests/shell/testcases/bitwise/dumps/0040mark_binop_3.json-nft	2023-11-22 13:50:12.114098356 +0100
+++ /tmp/nft-test.20231122-135045.550.28uKHu/test-tests-shell-testcases-bitwise-0040mark_binop_3.1/ruleset-after.json	2023-11-22 13:50:45.622065923 +0100
@@ -1 +1 @@
-{"nftables": [{"metainfo": {"version": "VERSION", "release_name": "RELEASE_NAME", "json_schema_version": 1}}, {"table": {"family": "ip", "name": "t", "handle": 0}}, {"chain": {"family": "ip", "table": "t", "name": "c", "handle": 0, "type": "filter", "hook": "output", "prio": 0, "policy": "accept"}}, {"rule": {"family": "ip", "table": "t", "chain": "c", "handle": 0, "expr": [{"mangle": {"key": {"ct": {"key": "mark"}}, "value": {"|": [{"<<": [{"payload": {"protocol": "ip", "field": "dscp"}}, 2]}, 16]}}}]}}]}
+{"nftables": [{"metainfo": {"version": "VERSION", "release_name": "RELEASE_NAME", "json_schema_version": 1}}, {"table": {"family": "ip", "name": "t", "handle": 0}}, {"chain": {"family": "ip", "table": "t", "name": "c", "handle": 0, "type": "filter", "hook": "input", "prio": 0, "policy": "accept"}}, {"rule": {"family": "ip", "table": "t", "chain": "c", "handle": 0, "expr": [{"mangle": {"key": {"meta": {"key": "mark"}}, "value": {"|": [{"<<": [{"payload": {"protocol": "ip", "field": "dscp"}}, 2]}, 16]}}}]}}]}
--- /dev/fd/63	2023-11-22 13:50:45.627065918 +0100
+++ /dev/fd/62	2023-11-22 13:50:45.627065918 +0100
@@ -21,7 +21,7 @@
         "name": "c",
         "handle": 0,
         "type": "filter",
-        "hook": "output",
+        "hook": "input",
         "prio": 0,
         "policy": "accept"
       }
@@ -36,7 +36,7 @@
           {
             "mangle": {
               "key": {
-                "ct": {
+                "meta": {
                   "key": "mark"
                 }
               },

Re: [PATCH nft 1/1] tests: prettify JSON in test output and add helper

[Pablo Neira Ayuso]

On Wed, Nov 22, 2023 at 01:55:57PM +0100, Thomas Haller wrote:
[...]
> As always, you will find some result files in /tmp/nft-
> test.latest.$USER/, which I usually read with
> 
>   $ grep --color=always ^ -aR /tmp/nft-test.latest.*/ | less -R
> 
> there will be a new file there, named "ruleset-diff.json.pretty".
> 
> It contains the content of "./tests/shell/helpers/json-diff-pretty.sh"
> output.
> 
> 
> For example:
> 
>   $ cp tests/shell/testcases/bitwise/dumps/0040mark_binop_2.json-nft tests/shell/testcases/bitwise/dumps/0040mark_binop_3.json-nft
>   $ ./tests/shell/run-tests.sh tests/shell/testcases/bitwise/0040mark_binop_3
> 
> 
> leaves a file
> 
>      /tmp/nft-test.latest.*/test-tests-shell-testcases-bitwise-0040mark_binop_3.1/ruleset-diff.json.pretty

Thanks for explaining, this is now applied.

And please, add copy and paste this info to the README file (unless I
am overlooking anything, it is not yet there).

Re: [PATCH nft 1/1] tests: prettify JSON in test output and add helper

[Thomas Haller]

On Wed, 2023-11-22 at 18:36 +0100, Pablo Neira Ayuso wrote:


> Thanks for explaining, this is now applied.

Thank you.

> 
> And please, add copy and paste this info to the README file (unless I
> am overlooking anything, it is not yet there).

I hear you. It will still happen!!


Thomas