[nft PATCH] tests/shell: Treat json-nft dumps as binary in git

[nft PATCH] tests/shell: Treat json-nft dumps as binary in git

[Phil Sutter]

The stored JSON dumps containing a single line of a thusand characters
in average mess up diffs in history and patches if they change. Mitigate
this by treating them as binary files.

In order to get useable diffs back, one may undo/override the attribute
within $GIT_DIR/info/attributes, preferrably by defining a custom diff
driver converting the single-line dumps into something digestable by
diff:

| # cat >>$GIT_DIR/info/attributes <<EOF
| tests/shell/testcases/**/dumps/*.json-nft diff=json-nft
| EOF
| # cat >>$GIT_DIR/config <<EOF
| [diff "json-nft"]
|         binary = true
|         textconv = tests/shell/helpers/json-pretty.sh
| EOF

Stating the obvious: The textconv tool is for display purposes only,
patches will still contain binary diffs for the files.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 .gitattributes                     | 1 +
 tests/shell/helpers/json-pretty.sh | 4 ++++
 2 files changed, 5 insertions(+)
 create mode 100644 .gitattributes

diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000000000..17d78d0554929
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1 @@
+tests/shell/testcases/**/dumps/*.json-nft binary
diff --git a/tests/shell/helpers/json-pretty.sh b/tests/shell/helpers/json-pretty.sh
index 0d6972b81e2f0..d773da2be29de 100755
--- a/tests/shell/helpers/json-pretty.sh
+++ b/tests/shell/helpers/json-pretty.sh
@@ -1,5 +1,9 @@
 #!/bin/bash -e
 
+# support being called with input
+# in a file specified on command line
+[ -f "$1" ] && exec $0 <"$1"
+
 # WARNING: the output is not guaranteed to be stable.
 
 if command -v jq &>/dev/null ; then
-- 
2.41.0

Re: [nft PATCH] tests/shell: Treat json-nft dumps as binary in git

[Florian Westphal]

Phil Sutter <phil@nwl.cc> wrote:
> The stored JSON dumps containing a single line of a thusand characters
> in average mess up diffs in history and patches if they change. Mitigate
> this by treating them as binary files.
> 
> In order to get useable diffs back, one may undo/override the attribute
> within $GIT_DIR/info/attributes, preferrably by defining a custom diff
> driver converting the single-line dumps into something digestable by
> diff:

I'd say we convert the single-line dumps to jq format one-by-one
if they need updating.

New dumps are no longer accepted in single-line format.

Re: [nft PATCH] tests/shell: Treat json-nft dumps as binary in git

[Pablo Neira Ayuso]

On Fri, Nov 24, 2023 at 09:49:51AM +0100, Florian Westphal wrote:
> Phil Sutter <phil@nwl.cc> wrote:
> > The stored JSON dumps containing a single line of a thusand characters
> > in average mess up diffs in history and patches if they change. Mitigate
> > this by treating them as binary files.
> > 
> > In order to get useable diffs back, one may undo/override the attribute
> > within $GIT_DIR/info/attributes, preferrably by defining a custom diff
> > driver converting the single-line dumps into something digestable by
> > diff:
> 
> I'd say we convert the single-line dumps to jq format one-by-one
> if they need updating.

I am also fine with .json-nft dumps in pretty format too, which is
friendlier to git diff.

Re: [nft PATCH] tests/shell: Treat json-nft dumps as binary in git

[Phil Sutter]

On Fri, Nov 24, 2023 at 10:33:50AM +0100, Pablo Neira Ayuso wrote:
> On Fri, Nov 24, 2023 at 09:49:51AM +0100, Florian Westphal wrote:
> > Phil Sutter <phil@nwl.cc> wrote:
> > > The stored JSON dumps containing a single line of a thusand characters
> > > in average mess up diffs in history and patches if they change. Mitigate
> > > this by treating them as binary files.
> > > 
> > > In order to get useable diffs back, one may undo/override the attribute
> > > within $GIT_DIR/info/attributes, preferrably by defining a custom diff
> > > driver converting the single-line dumps into something digestable by
> > > diff:
> > 
> > I'd say we convert the single-line dumps to jq format one-by-one
> > if they need updating.

This might work without a transitioning mechanism if jq output piped
through jq does not change. The testsuite could just pipe the dump
through jq before comparing regardless of whether it's pretty-printed
already or not.

> I am also fine with .json-nft dumps in pretty format too, which is
> friendlier to git diff.

Discuss that with Thomas, please. I tried and failed, alleviating the
effects is my last resort.

Cheers, Phil

Re: [nft PATCH] tests/shell: Treat json-nft dumps as binary in git

[Thomas Haller]

On Fri, 2023-11-24 at 11:02 +0100, Phil Sutter wrote:
> On Fri, Nov 24, 2023 at 10:33:50AM +0100, Pablo Neira Ayuso wrote:
> > On Fri, Nov 24, 2023 at 09:49:51AM +0100, Florian Westphal wrote:
> > > Phil Sutter <phil@nwl.cc> wrote:
> > > > The stored JSON dumps containing a single line of a thusand
> > > > characters
> > > > in average mess up diffs in history and patches if they change.
> > > > Mitigate
> > > > this by treating them as binary files.
> > > > 
> > > > In order to get useable diffs back, one may undo/override the
> > > > attribute
> > > > within $GIT_DIR/info/attributes, preferrably by defining a
> > > > custom diff
> > > > driver converting the single-line dumps into something
> > > > digestable by
> > > > diff:
> > > 
> > > I'd say we convert the single-line dumps to jq format one-by-one
> > > if they need updating.
> 
> This might work without a transitioning mechanism if jq output piped
> through jq does not change. The testsuite could just pipe the dump
> through jq before comparing regardless of whether it's pretty-printed
> already or not.
> 
> > I am also fine with .json-nft dumps in pretty format too, which is
> > friendlier to git diff.
> 
> Discuss that with Thomas, please. I tried and failed, alleviating the
> effects is my last resort.
> 
> Cheers, Phil
> 

Hi,

let me send a patch, that hopefully makes you all happy.
(or at least, something to discuss further).

Hold on.

Thomas