git diff vs git diff-files

git diff vs git diff-files

[Bernd Jendrissek]

I have a billion testsuite golden output files that have changed due
to an explicit ordering of objects I've imposed on output. A helper
script I wrote to help parse the diffs (to ignore order-only
differences) noticed that one hunk had a different number of additions
and deletions. I'm manually copying and pasting the hunks from git add
-i into the script. However, when I used git diff to get at the
changes, to discover which line was the offending one, all additions
and deletions were exactly matched!

With strace I noticed that git add -i calls git diff-files, and that's
about as far as I can conveniently trace where things are going weird.
Weird:

$ /usr/local/git/bin/git diff-files -p --color -- TwoStageAmp-output.net
diff --git a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
index a5ee1e7..a9f3620 100644
--- a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
+++ b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
@@ -1,47 +1,47 @@
-Part unknown { Name Cout }
-Part unknown { Name R5 }
-Part unknown { Name R4 }
-Part unknown { Name RE2 }
-Part unknown { Name Q2 }
+Part unknown { Name R8 }
 Part unknown { Name A3 }
-Part unknown { Name R3 }
 Part unknown { Name A2 }
-Part unknown { Name RE1 }
 Part unknown { Name A1 }
[snip]

$ /usr/local/git/bin/git diff TwoStageAmp-output.netdiff --git
a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.n
index a5ee1e7..a9f3620 100644
--- a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
+++ b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
@@ -1,47 +1,47 @@
-Part unknown { Name Cout }
-Part unknown { Name R5 }
-Part unknown { Name R4 }
-Part unknown { Name RE2 }
-Part unknown { Name Q2 }
+Part unknown { Name R8 }
 Part unknown { Name A3 }
-Part unknown { Name R3 }
 Part unknown { Name A2 }
-Part unknown { Name RE1 }
-Part unknown { Name Q1 }  <--- Not present with diff-files output
 Part unknown { Name A1 }
[snip]

The index is clean on TwoStageAmp-output.net (but by now a million
other files are dirty in the index). While I discovered this issue
with Ubuntu's 1.7.1, I got the same behaviour with git
1.7.12.rc2.18.g61b472e

What's going on here? I'll leave my tree in whatever weird state it
might be, for a while, in case anyone wants me to try various things.
At this point I want to understand what's going on, in case there's
either a learning moment for me or a bug, rather than just finding a
workaround.

Re: git diff vs git diff-files

[Thomas Rast]

Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:

> $ /usr/local/git/bin/git diff-files -p --color -- TwoStageAmp-output.net
> diff --git a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> index a5ee1e7..a9f3620 100644
> --- a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> +++ b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> @@ -1,47 +1,47 @@
> -Part unknown { Name Cout }
> -Part unknown { Name R5 }
> -Part unknown { Name R4 }
> -Part unknown { Name RE2 }
> -Part unknown { Name Q2 }
> +Part unknown { Name R8 }
>  Part unknown { Name A3 }
> -Part unknown { Name R3 }
>  Part unknown { Name A2 }
> -Part unknown { Name RE1 }
>  Part unknown { Name A1 }
> [snip]
>
> $ /usr/local/git/bin/git diff TwoStageAmp-output.netdiff --git
> a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.n
> index a5ee1e7..a9f3620 100644
> --- a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> +++ b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> @@ -1,47 +1,47 @@
> -Part unknown { Name Cout }
> -Part unknown { Name R5 }
> -Part unknown { Name R4 }
> -Part unknown { Name RE2 }
> -Part unknown { Name Q2 }
> +Part unknown { Name R8 }
>  Part unknown { Name A3 }
> -Part unknown { Name R3 }
>  Part unknown { Name A2 }
> -Part unknown { Name RE1 }
> -Part unknown { Name Q1 }  <--- Not present with diff-files output
>  Part unknown { Name A1 }
> [snip]

Can you share this repository?  Or at least the pre- and post-change
files, transferred in such a way that there won't be any whitespace
damage (your snippets above show obvious damage).  You can use

  git cat-file blob :gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net

to get at the contents of the version in the index.

Do you have any diff config that could be of interest?  A textconv
filter would be an obvious example that could produce the above, but
perhaps you could just look at

  git config --get-regexp '.*diff.*'
  grep diff .gitattributes .git/info/attributes

etc.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: git diff vs git diff-files

[Matthieu Moy]

Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:

> $ /usr/local/git/bin/git diff-files -p --color -- TwoStageAmp-output.net
> diff --git a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> index a5ee1e7..a9f3620 100644

> $ /usr/local/git/bin/git diff TwoStageAmp-output.netdiff --git
> a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
> b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.n
> index a5ee1e7..a9f3620 100644

What's surprising is that both diff show the same "index" line, so both
commands actually diff the same content, and then show a different
output.

You can try something like

git cat-file blob a5ee1e7 > /tmp/staged.txt
diff -u /tmp/staged.txt TwoStageAmp-output.netdiff

to recover the content from the index (Thomas' version should also work
and give the same result), and use another diff tool. This "diff -u"
should give you an output similar to one of "git diff" and "git
diff-files" (my understanding is that it should match diff-files, and
"git diff" is the one being wrong here).

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: git diff vs git diff-files

[Bernd Jendrissek]

On Mon, Aug 13, 2012 at 5:02 PM, Thomas Rast <trast@student.ethz.ch> wrote:
> Can you share this repository?

This weird behaviour doesn't even survive making a copy (cp -a) of the
whole repository, so I very much doubt making it available would be
illuminative. My disk's SMART data seems okay. The weird-quotient just
rose a bit.

Besides, .git/ is 60MB and my upload speed is 128kbps. A bit inconvenient.

> Or at least the pre- and post-change
> files, transferred in such a way that there won't be any whitespace
> damage (your snippets above show obvious damage).  You can use

http://www.bpj-code.co.za/downloads.php/bugs/TwoStageAmp-output.net?text
contains the output from git show a5ee1e7. Leave off the ?text for an
application/octet-stream download.

That file (with the Q1 line present) is consistent with the earlier
commit that added the file. It's diff-files that's lying.

> Do you have any diff config that could be of interest?  A textconv
> filter would be an obvious example that could produce the above, but
> perhaps you could just look at

Nothing that fancy. I have just diff.color = auto and user.* = blah in
global config, and similarly benign config in the repository.

Re: git diff vs git diff-files

[Thomas Rast]

Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:

> On Mon, Aug 13, 2012 at 5:02 PM, Thomas Rast <trast@student.ethz.ch> wrote:
>> Can you share this repository?
>
> This weird behaviour doesn't even survive making a copy (cp -a) of the
> whole repository, so I very much doubt making it available would be
> illuminative. My disk's SMART data seems okay. The weird-quotient just
> rose a bit.

That sounds like an issue with stat() data, and then it doesn't.  cp -a
would change all the inode numbers, triggering a full refresh of the
index in 'git diff'.  But I'm not sure exactly how this can lead to
disagreement, since both commands read *both* index and worktree version
of the file.  Very confusing.

Can you try the following:

  git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
  cp .git/index .git/index.orig
  touch gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
  # note, it is important that you run diff first
  git diff gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
  git diff-files -p gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
  git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net

>> Or at least the pre- and post-change
>> files, transferred in such a way that there won't be any whitespace
>> damage (your snippets above show obvious damage).  You can use
>
> http://www.bpj-code.co.za/downloads.php/bugs/TwoStageAmp-output.net?text
> contains the output from git show a5ee1e7. Leave off the ?text for an
> application/octet-stream download.

Umm, that's only one side of the diff, isn't it?  The hunk header claims
that the hunk goes up to line 47, so there must be more changes beyond
what you showed.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: git diff vs git diff-files

[Bernd Jendrissek]

On Tue, Aug 14, 2012 at 10:18 AM, Thomas Rast <trast@student.ethz.ch> wrote:
> That sounds like an issue with stat() data, and then it doesn't.  cp -a
> would change all the inode numbers, triggering a full refresh of the
> index in 'git diff'.  But I'm not sure exactly how this can lead to
> disagreement, since both commands read *both* index and worktree version
> of the file.  Very confusing.
>
> Can you try the following:
>
>   git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
>   cp .git/index .git/index.orig
>   touch gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
>   # note, it is important that you run diff first
>   git diff gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
>   git diff-files -p gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
>   git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net

I'm can't tell what this was supposed to prove, but I think it's
proven that I'm in the wrong mailing list, and that I should be taking
this up with gnome-terminal. The Q1 part deletion line seems to be
appearing (in the terminal) only after I've copy&pasted it into
another terminal.

Running the same commands in xterm (correctly) shows the Q1 part
deletion line without having to copy & paste.

At this point I doubt there's much value in publishing the output, but
just in case (and it may be relevant when i take it to
gnome-terminal), it's here:

http://www.bpj-code.co.za/downloads.php/bugs/git-output-87vcgl999v.fsf?text

Note that line 104 of that output appears in the same terminal from
which I copy&paste it only after I switch back to it, after having
pasted the output. Oi vey.

>> http://www.bpj-code.co.za/downloads.php/bugs/TwoStageAmp-output.net?text
>> contains the output from git show a5ee1e7. Leave off the ?text for an
>> application/octet-stream download.
>
> Umm, that's only one side of the diff, isn't it?  The hunk header claims
> that the hunk goes up to line 47, so there must be more changes beyond
> what you showed.

You're right - the hunk isn't even self-consistent!
http://www.bpj-code.co.za/downloads.php/bugs/git-add--p-TwoStageAmp-output.net?text
is the "bad" diff, and I count only 46 lines of (context + deletions),
while the hunk header claims the (correct) 47 lines corresponding to
the whole file covered in the hunk.

Thanks for your time; I'm fairly convinced now that git is just an
innocent accessory to another program's bug.

Re: git diff vs git diff-files

[Thomas Rast]

Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:

> On Tue, Aug 14, 2012 at 10:18 AM, Thomas Rast <trast@student.ethz.ch> wrote:
>> Can you try the following:
>>
>>   git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
>>   cp .git/index .git/index.orig
>>   touch gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
>>   # note, it is important that you run diff first
>>   git diff gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
>>   git diff-files -p gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
>>   git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
>
> I'm can't tell what this was supposed to prove, but I think it's
> proven that I'm in the wrong mailing list, and that I should be taking
> this up with gnome-terminal. The Q1 part deletion line seems to be
> appearing (in the terminal) only after I've copy&pasted it into
> another terminal.
>
> Running the same commands in xterm (correctly) shows the Q1 part
> deletion line without having to copy & paste.

Fun stuff.  This will be very interesting to debug.  It's still
possible, though perhaps not extremely likely, that git is writing
garbage to the terminal and it just happens to work for xterm.

Whatever you do next, *please* post the *exact* file contents *and*
output in a format that does not suffer any transport damage, neither to
whitespace nor to binary data (such as terminal escapes).  Piping
through xxd comes to mind, perhaps by using

  GIT_PAGER=xxd git -p diff ....  >diff-hexdump

etc. to ensure that git takes the same code paths as when writing to
less.

Cut&paste from a terminal will at best give you exactly what the
terminal figures it is *currently displaying*, which is completely
different from what programs write to it.  At worst, it will also suffer
whitespace damage.  So pretty much everything you showed so far, except
for the raw data of one (why only one?!) side of the diff, is useless.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: git diff vs git diff-files

[Andreas Schwab]

Thomas Rast <trast@student.ethz.ch> writes:

> Whatever you do next, *please* post the *exact* file contents *and*
> output in a format that does not suffer any transport damage, neither to
> whitespace nor to binary data (such as terminal escapes).  Piping
> through xxd comes to mind, perhaps by using
>
>   GIT_PAGER=xxd git -p diff ....  >diff-hexdump
>
> etc. to ensure that git takes the same code paths as when writing to
> less.

Or use script(1).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: git diff vs git diff-files

[Bernd Jendrissek]

On Tue, Aug 14, 2012 at 11:55 AM, Thomas Rast <trast@student.ethz.ch> wrote:
> Fun stuff.  This will be very interesting to debug.  It's still
> possible, though perhaps not extremely likely, that git is writing
> garbage to the terminal and it just happens to work for xterm.
>
> Whatever you do next, *please* post the *exact* file contents *and*

Hmm, this really isn't about git anymore, as I've reduced this problem
to the following file:

0000000: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .
0000010: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .
0000020: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .
0000030: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .
0000040: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .
0000050: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .
0000060: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .
0000070: 200a 200a 200a 200a 200a 200a             . . . . . .

berndj@capybara:~/gaf.git$ nl /tmp/g
     2	
     3	
     4	
     5	
     6	
     7
[snip]
    60	
    61	
    62	
berndj@capybara:~/gaf.git$

Anything shorter than that causes first the command line (what you
might call line 0, not the one you might call line 63, and yes, I do
not that this is close to a binary round number), and after that,
nothing to disappear.

strace confirms that gnome-terminal reads, but neglects to write, line 1.

I notice now that a second gnome-terminal tab in the same directory
running the same commands reliably displays the correct output. I must
have dumped an ELF file to the terminal at some point, or something
similarly daft like that, and forgotten to reset the terminal. Not
that I recall having done such a thing recently.

In any case, it's too late now; I ran reset(1) and the terminal is
back behaving itself.

Thanks for helping me get (close to) the bottom of this. If git were
to be guilty (which I very much doubt), the evidence is long gone now.
I'll chalk this up to cosmic rays or something.