end-of-line diff checkout direction dependence problem

end-of-line diff checkout direction dependence problem

[Thomas Vieten]

We face a very inconvenient problem with end-of-line diffs which are not 
"real".
We know the end-of-line problem very well as we thought.
But now we found a new phenomenon and nobody mentioning it.

Consider the following repository structure:

           -----------|----|------------->branch1
         /
master
         \
----------|-------|---------|--------------->branch2

The branches are based on master/head.
We just consider one branch here, e.g. branch1 .

Working with the head of branch1 gives no problems. No end-of-line diffs.
Also going back in the direction of master - no problems.
Only in the case if we do a checkout from branch1 to master, then
all of a sudden end-of-line diffs appear.
The files might be changed, but the end-of-line attributes in 
gitattributes are not changed in the branch.

It seems to be the case that since the last change to the files which 
pop up with eol differences, gittattributes where changed and touch 
their extensions.

With the operation

git ls-files -z | xargs -0 rm -f  # delete all the files of this version 
- here master/head
git reset --hard                  # force checkout of master/head and 
reset index

The problem disappears! No eol problems anymore. Something like a brute 
force checkout.

Also checking out versions in the direction of branch1 give never 
end-of-line diffs.

What has changed somewhen are the gitattributes.

We estimate that this becomes a problem when applying the diffs from 
branch1 in the direction of
the master. Finally then the diffs result in a different state from the 
master.

But when the master is checked out freshly, this difference does not appear.

Very, very annoying.

We check now every time when these end-of-line diffs appear, if they are 
really end of line diffs

git diff --ignore-space-at-eol

and then try the procedure above.

But to have a dependence from the direction of the checkout is somewhat 
irritating.

If this is not a bug - then how to avoid it ?

bye for now

Thomas

-- 
++++++++++++++++++++++++++++++++++++++++++++++++++++++
iVISION GmbH		|Industrial Inspection Systems
Jülicherstr. 336 B	|
D-52070 Aachen		|
Tel: +49-(0)241-961233	|
FAX: +49-(0)241-980 2064|
www.ivision.de
tvie@ivision.de
++++++++++++++++++++++++++++++++++++++++++++++++++++++

Re: end-of-line diff checkout direction dependence problem

[Torsten Bögershausen]

On 2015-06-30 16.12, Thomas Vieten wrote:
> We face a very inconvenient problem with end-of-line diffs which are not "real".
> We know the end-of-line problem very well as we thought.
> But now we found a new phenomenon and nobody mentioning it.
> 
> Consider the following repository structure:
> 
>           -----------|----|------------->branch1
>         /
> master
>         \
> ----------|-------|---------|--------------->branch2
> 
> The branches are based on master/head.
> We just consider one branch here, e.g. branch1 .
> 
> Working with the head of branch1 gives no problems. No end-of-line diffs.
> Also going back in the direction of master - no problems.
> Only in the case if we do a checkout from branch1 to master, then
> all of a sudden end-of-line diffs appear.
> The files might be changed, but the end-of-line attributes in gitattributes are
> not changed in the branch.
> 
> It seems to be the case that since the last change to the files which pop up
> with eol differences, gittattributes where changed and touch their extensions.
> 
> With the operation
> 
> git ls-files -z | xargs -0 rm -f  # delete all the files of this version - here
> master/head
> git reset --hard                  # force checkout of master/head and reset index
> 
> The problem disappears! No eol problems anymore. Something like a brute force
> checkout.
> 
> Also checking out versions in the direction of branch1 give never end-of-line
> diffs.
> 
> What has changed somewhen are the gitattributes.
> 
> We estimate that this becomes a problem when applying the diffs from branch1 in
> the direction of
> the master. Finally then the diffs result in a different state from the master.
> 
> But when the master is checked out freshly, this difference does not appear.
> 
> Very, very annoying.
> 
> We check now every time when these end-of-line diffs appear, if they are really
> end of line diffs
> 
> git diff --ignore-space-at-eol
> 
> and then try the procedure above.
> 
> But to have a dependence from the direction of the checkout is somewhat irritating.
> 
> If this is not a bug - then how to avoid it ?
> 
> bye for now
> 
> Thomas
> 
The things which are described don't sound unfamilar.
First some questions:
Which Git/OS are you running on ?

CYGWIN ?
Git-for-Windows ?
Linux ?
Other ?

Which versions ?
How does your .gitattribute file look like ?

It may be, that you need to "nornalize" your repo:

https://www.kernel.org/pub/software/scm/git/docs/gitattributes.html
The search for this text
"When text=auto normalization"
and follow the instructions:

Re: end-of-line diff checkout direction dependence problem

[Thomas Vieten]

Answers and new response below.

Thomas Vieten wrote:
> On 2015-06-30 16.12, Thomas Vieten wrote:
>> We face a very inconvenient problem with end-of-line diffs which are 
>> not "real".
>> We know the end-of-line problem very well as we thought.
>> But now we found a new phenomenon and nobody mentioning it.
>>
>> Consider the following repository structure:
>>
>>           -----------|----|------------->branch1
>>         /
>> master
>>         \
>> ----------|-------|---------|--------------->branch2
>>
>> The branches are based on master/head.
>> We just consider one branch here, e.g. branch1 .
>>
>> Working with the head of branch1 gives no problems. No end-of-line 
>> diffs.
>> Also going back in the direction of master - no problems.
>> Only in the case if we do a checkout from branch1 to master, then
>> all of a sudden end-of-line diffs appear.
>> The files might be changed, but the end-of-line attributes in 
>> gitattributes are
>> not changed in the branch.
>>
>> It seems to be the case that since the last change to the files which 
>> pop up
>> with eol differences, gittattributes where changed and touch their 
>> extensions.
>>
>> With the operation
>>
>> git ls-files -z | xargs -0 rm -f  # delete all the files of this 
>> version - here
>> master/head
>> git reset --hard                  # force checkout of master/head and 
>> reset index
>>
>> The problem disappears! No eol problems anymore. Something like a 
>> brute force
>> checkout.
>>
>> Also checking out versions in the direction of branch1 give never 
>> end-of-line
>> diffs.
>>
>> What has changed somewhen are the gitattributes.
>>
>> We estimate that this becomes a problem when applying the diffs from 
>> branch1 in
>> the direction of
>> the master. Finally then the diffs result in a different state from 
>> the master.
>>
>> But when the master is checked out freshly, this difference does not 
>> appear.
>>
>> Very, very annoying.
>>
>> We check now every time when these end-of-line diffs appear, if they 
>> are really
>> end of line diffs
>>
>> git diff --ignore-space-at-eol
>>
>> and then try the procedure above.
>>
>> But to have a dependence from the direction of the checkout is 
>> somewhat irritating.
>>
>> If this is not a bug - then how to avoid it ?
>>
>> bye for now
>>
>> Thomas
>>
> The things which are described don't sound unfamilar.
> First some questions:
> Which Git/OS are you running on ?
>
> CYGWIN ?
> Git-for-Windows ?
git version 1.9.2.msysgit.0
+tortoisegit, for viewing and conflicts
> Linux ?
> Other ?
Windows 7 Pro
>
> Which versions ?
> How does your .gitattribute file look like ?
>
see the file attachend to the end of this message
> It may be, that you need to "nornalize" your repo:
>
> https://www.kernel.org/pub/software/scm/git/docs/gitattributes.html
> The search for this text
> "When text=auto normalization"
> and follow the instructions:
>
in principle we know all this.
What is remarkable that we are able to checkout a version of master 
which is not consistent with the repo, and more, dependent from the 
checkout direction (if the direction is the positive or negative history 
in time). And on the other hand we can checkout a version of master 
which is in sync with the master.

Normally such conflicts with not normalised repos appear immediately 
also in positive history direction. And then it is possible to detect them.
The other way around - negative history and diffs - it causes a big 
questionmark.

On the other hand this would lead to the mandatory work flow advice: 
"Always normalize the repository after changes within the gitattributes 
file"
And then: Should this then not be automatically be done somehow in the 
background by git ?
Reasoning: if the "git machine" is causing this behaviour 
systematically, shouldn't the machine itself have compensation, correction?

Depending on your point of view this could be seen as a bug.

There is also a big question open: will normalisation really help ? 
Because there must be one commit with the new gitattributes and then you 
normalize.
But the "wrong diff" is in the repo and will cause the problem when 
going back to master in the negative direction.
This is how understand it up to now.

But at this point git is complex and we are not really the experts.

best regards

Thomas V.


--------------------------------------------------------------------------------------------------------
.gitattributes
#Set default behaviour, in case users don't have core.autocrlf set.
* text=auto
*.PC                text
*.in                text
*.hex               text
*.inc               text
*.DAT               binary
*.lic               binary
*.chm               binary
*.dat               binary
*.cam               binary
*.cam.new           binary
*.cam.cam           binary
*.am                text
*.cc                text
*.m4                eol=lf
*.exe               binary
*.sys               binary
*.dll               binary
*.cxf               eol=lf
*.vcproj            eol=crlf
*.vcxproj           text
*.vcxproj.filters   text
*.sln               eol=crlf
*.dsp               eol=crlf
*.dsw               eol=crlf
*.sh                text
*.txt               text
*.png               binary
*.jpg               binary
*.tif               binary
*.odt               binary
*.db                binary
*.vsd               binary
*.dbf               binary
*.zip               binary
*.emf               binary
*.psp               binary
*.dot               binary
*.doc               diff=word
*.sql               text
*.vsd               binary
*.sys               binary
*.xls               binary
*.vxd               binary
*.jpg               binary
*.jpeg              binary
*.sch               binary
*.pdf               binary
*.s#*               binary
*.ocx               binary
*.gif               binary
*.gz                binary
*.obj               binary
*.ncb               binary
*.bin               binary
*.bmp               binary
*.cmd               eol=crlf
*.bat               eol=crlf
*.htm               eol=crlf
*.html              eol=crlf
*.dsp               eol=crlf
*.dbf               binary
*.vtg               binary
*.mdb               binary
*.msi               binary
*.cbproj            binary
*.groupproj         binary
*.bdsgroup          binary
*.tgr               binary
*.pbd               binary
*.pbl               binary
*.tag               binary
makefile            text
*.c                 text
*.cpp               text
*.h                 text
*.hpp               text

Re: end-of-line diff checkout direction dependence problem

[Torsten Bögershausen]

On 2015-07-02 16.00, Thomas Vieten wrote:
[]
> see the file attachend to the end of this message
Thanks for the info
>
>> It may be, that you need to "nornalize" your repo:
>>
> in principle we know all this.
> What is remarkable that we are able to checkout a version of master which is
> not consistent with the repo, and more, dependent from the checkout direction
> (if the direction is the positive or negative history in time). And on the
> other hand we can checkout a version of master which is in sync with the master.
>
> Normally such conflicts with not normalised repos appear immediately also in
> positive history direction. And then it is possible to detect them.
> The other way around - negative history and diffs - it causes a big questionmark.
>
> On the other hand this would lead to the mandatory work flow advice: "Always
> normalize the repository after changes within the gitattributes file"
Yes, I think this sounds reasonable, and I think the documentations says this.
But it is a manual step, which is typically done only once.
It there anything which can be improved here ?
> And then: Should this then not be automatically be done somehow in the
> background by git ?
This could make sense, but the word "background" should mean visible to the user ?
Recently we got the "untracked cache" into Git, which keeps track about the
.gitignore files
A similar logic can be used to keep track of the .gitattributes file(s) (There
can be more than one)
Patches are welcome.
>
> Reasoning: if the "git machine" is causing this behaviour systematically,
> shouldn't the machine itself have compensation, correction?
>
Question 1: The documentation should be clear enough:
 Whenever someone "introduces .gitattributes," the repo should be normalized.

> Depending on your point of view this could be seen as a bug.
I have run into this myself, but never in a reproducable way.
The day I can reproduce it, I may send a patch.
Or, somebody else sends a patch before that day.
>
> There is also a big question open: will normalisation really help ? Because
> there must be one commit with the new gitattributes and then you normalize.
To my experience it does.
 Normalizing means one single commit (not 2), including the .gitattributes and
the normalized files.
Question 2:
Is there something in the documentation that could be improved.
> But the "wrong diff" is in the repo and will cause the problem when going back
> to master in the negative direction.
> This is how understand it up to now.
>
> But at this point git is complex and we are not really the experts.
>
> best regards
>
> Thomas V.
>
Thanks for the report(s)