core.autocrlf & Cygwin - files incorrectly flagged as modified

core.autocrlf & Cygwin - files incorrectly flagged as modified

[David Antliff]

I approached the Cygwin mailing list about this problem, and the package
maintainer suggested I ask here as well, in case someone can shed some light on
this.

I am using git-1.6.4.2 (as part of Cygwin-1.7 beta) and I've observed some
behaviour that seems wrong to me.

When we deployed git on Cygwin, we decided to set core.autocrlf=true, as many of
our users are working with DOS/Windows files (CRLF endings). Some users are
using Linux and working on the same code. I'm not actually sure why we chose to
use this setting - there's a lot of conflicting advice around. I think the final
decision was made so that kdiff3 would operate correctly. However we did go to
lengths to ensure that everyone is using the same configuration setting.

The problem is that sometimes, after a git-clone, the output of git-status and
git-diff shows entire files as being different. However these files have not
been modified by the user - only git has had a chance to change them (due to
autocrlf=true). But surely if git has converted the file automatically, it
should know that it has to compensate for this when comparing with the local
repository?

And I think it usually does this and almost all files behave properly. The
problem arises when the source-controlled file contains lines with trailing
whitespace. In this situation, git-diff clearly indicates a difference and this
blocks merges and checkouts.

I suspect what is happening is that the line conversion routine in git might be
stripping trailing whitespace, as well as converting the line endings. This
operation is not properly accounted for in the reverse direction, and the file
is flagged as modified.

The Cygwin git package maintainer has indicated that he thinks this might be a
bug: http://cygwin.com/ml/cygwin/2009-12/msg00466.html

Also, as cloned files are converted to DOS-line-endings, by default Cygwin's
bash cannot run any scripts as they have the wrong line endings. I have to set
the 'permanent' bash variable SHELLOPTS to include 'igncr' before bash scripts
can run. Perhaps this is wrong and git on Cygwin (with binary mounts) should be
converting to UNIX line endings instead?

At one point I tried switching off core.autocrlf for myself but this caused a
lot of conflicts due to mismatched line-endings. It seems to me that if we want
to switch to this, *everyone* has to do it at once.

I'd like to know whether this is a problem worthy of further investigation,
whether it should fall in the git or Cygwin camp, and whether I really ought to
be using autocrlf=true please.

Re: core.autocrlf & Cygwin - files incorrectly flagged as modified

[Tay Ray Chuan]

Hi,

On Wed, Dec 16, 2009 at 6:24 AM, David Antliff <david.antliff@gmail.com> wrote:
> I suspect what is happening is that the line conversion routine in git might be
> stripping trailing whitespace, as well as converting the line endings. This
> operation is not properly accounted for in the reverse direction, and the file
> is flagged as modified.

I am not a Git expert, but could it be your editor on Windows?

Not just stripping whitespace; it might be setting the eol to LF
(instead of CRLF) when saving the file.

> Also, as cloned files are converted to DOS-line-endings, by default Cygwin's
> bash cannot run any scripts as they have the wrong line endings. I have to set
> the 'permanent' bash variable SHELLOPTS to include 'igncr' before bash scripts
> can run. Perhaps this is wrong and git on Cygwin (with binary mounts) should be
> converting to UNIX line endings instead?

Again, IANAGE, but according to the manual, this should be expected,
because Git, when writing to the filesystem, converts LF to CRLF:

  If true, makes git convert CRLF at the end of lines in text files to
  LF when reading from the filesystem, and convert in reverse when
  writing to the filesystem.

> At one point I tried switching off core.autocrlf for myself but this caused a
> lot of conflicts due to mismatched line-endings. It seems to me that if we want
> to switch to this, *everyone* has to do it at once.

Just make this conversion a commit, and depending on your project's
workflow, push/pull it.

Although the whole file will be marked as changed, you can always
double-check that only whitespace changes have been made by running
git-diff with --ignore-space-at-eol or -b.

-- 
Cheers,
Ray Chuan

Re: core.autocrlf & Cygwin - files incorrectly flagged as modified

[David Antliff]

Tay Ray Chuan writes:
> On Wed, Dec 16, 2009 at 6:24 AM, David Antliff wrote:
> > I suspect what is happening is that the line conversion routine in git
> > might be
> > stripping trailing whitespace, as well as converting the line endings. This
> > operation is not properly accounted for in the reverse direction, and the
> > file is flagged as modified.
> 
> I am not a Git expert, but could it be your editor on Windows?

Hi Ray, thank you for responding.

This observation is made immediately after the clone, before any editor (or any
other program) even touches the files. If you can imagine doing a fresh clone
and then immediately doing 'git-status' and seeing modified files, you'd have a
pretty good idea of what I'm seeing. Git itself has somehow marked the files as
modified on check-out. I am fairly certain it's related to trailing whitespace
on one or more lines within the affected files.

If there's no trailing whitespace, git converts the file to CRLF on checkout
and shows no modification (since it knows to convert back when referring to
local repository).

> > Also, as cloned files are converted to DOS-line-endings, by default Cygwins
> > bash cannot run any scripts as they have the wrong line endings. I have to 
> > set the 'permanent' bash variable SHELLOPTS to include 'igncr' before bash
> > scripts can run. Perhaps this is wrong and git on Cygwin (with binary 
> > mounts) should be converting to UNIX line endings instead?
> 
> Again, IANAGE, but according to the manual, this should be expected,
> because Git, when writing to the filesystem, converts LF to CRLF:
> 
>   If true, makes git convert CRLF at the end of lines in text files to
>   LF when reading from the filesystem, and convert in reverse when
>   writing to the filesystem.

Yes, perhaps the advice to use autocrlf=true on Cygwin (in binary mount mode,
since in text mode git doesn't work *at all*) is misplaced since Cygwin is not
expecting CRLF endings in that mode. Apparently it's required for apps like
Kdiff3 however... all so confusing really...

The inability to run bash scripts straight out of git when using autocrlf=true
is almost enough to suggest to me that this mode really shouldn't be used in
Cygwin.
 
> > At one point I tried switching off core.autocrlf for myself but this
> > caused a lot of conflicts due to mismatched line-endings. It seems to me
> > that if we want to switch to this, *everyone* has to do it at once.
> 
> Just make this conversion a commit, and depending on your project's
> workflow, push/pull it.

Yes, I think that's how it has to be done. Unfortunately I have 20+ projects
each with many active branches. The entire conversion is possible but it's
going to take a while... :)

> Although the whole file will be marked as changed, you can always
> double-check that only whitespace changes have been made by running
> git-diff with --ignore-space-at-eol or -b.

That's a helpful option - thank you.


I'd be quite interested to know if people are successfully using git in Cygwin
with autocrlf=true, and if so how they are working around these modified-files
and bash-compatibility problems.

Also, is anyone using autocrlf=false in Cygwin successfully?

Thanks again for your reply.

-- David.

Re: core.autocrlf & Cygwin - files incorrectly flagged as modified

[Robin Rosenberg]

tisdag 15 december 2009 23:24:15 skrev  David Antliff:
> I approached the Cygwin mailing list about this problem, and the package
> maintainer suggested I ask here as well, in case someone can shed some
>  light on this.
> 
> I am using git-1.6.4.2 (as part of Cygwin-1.7 beta) and I've observed some
> behaviour that seems wrong to me.
> 
> When we deployed git on Cygwin, we decided to set core.autocrlf=true, as
>  many of our users are working with DOS/Windows files (CRLF endings). Some
>  users are using Linux and working on the same code. I'm not actually sure
>  why we chose to use this setting - there's a lot of conflicting advice
>  around. I think the final decision was made so that kdiff3 would operate
>  correctly. However we did go to lengths to ensure that everyone is using
>  the same configuration setting.
> 
> The problem is that sometimes, after a git-clone, the output of git-status
>  and git-diff shows entire files as being different. However these files
>  have not been modified by the user - only git has had a chance to change
>  them (due to autocrlf=true). But surely if git has converted the file
>  automatically, it should know that it has to compensate for this when
>  comparing with the local repository?

AFAIK, this happens if you have CRLF line endings in the blobs in the repo.

-- robin

Re: core.autocrlf & Cygwin - files incorrectly flagged as modified

[David Antliff]

Robin Rosenberg writes:
> tisdag 15 december 2009 23:24:15 skrev  David Antliff:
> > The problem is that sometimes, after a git-clone, the output of git-status
> >  and git-diff shows entire files as being different. However these files
> >  have not been modified by the user - only git has had a chance to change
> >  them (due to autocrlf=true). But surely if git has converted the file
> >  automatically, it should know that it has to compensate for this when
> >  comparing with the local repository?
> 
> AFAIK, this happens if you have CRLF line endings in the blobs in the repo.

Oh? That could very well be true - how would such line endings get into a blob?
I'm pretty sure everyone is using autocrlf=true which should convert all line
endings to LF on commit, but perhaps there's another way to get CRLF line
endings into a blob?

If autocrlf=false, files are committed as-is (I believe), so this would be one
way to end up committing such files. But as far as I know, nobody has turned
that option off in our group.

Re: core.autocrlf & Cygwin - files incorrectly flagged as modified

[Robin Rosenberg]

fredagen den 18 december 2009 03.43.05 skrev  David Antliff:
> Robin Rosenberg writes:
> > tisdag 15 december 2009 23:24:15 skrev  David Antliff:
> > > The problem is that sometimes, after a git-clone, the output of
> > > git-status and git-diff shows entire files as being different. However
> > > these files have not been modified by the user - only git has had a
> > > chance to change them (due to autocrlf=true). But surely if git has
> > > converted the file automatically, it should know that it has to
> > > compensate for this when comparing with the local repository?
> >
> > AFAIK, this happens if you have CRLF line endings in the blobs in the
> > repo.
> 
> Oh? That could very well be true - how would such line endings get into a
>  blob? I'm pretty sure everyone is using autocrlf=true which should convert
>  all line endings to LF on commit, but perhaps there's another way to get
>  CRLF line endings into a blob?
> 
> If autocrlf=false, files are committed as-is (I believe), so this would be
>  one way to end up committing such files. But as far as I know, nobody has
>  turned that option off in our group.

EGit works this way at the moment. 
Conversion tools might do this.
Some Eclipse plugins produce files with CRLF regardless on platform and on
non-Windows the core.autocrlf flags is false by default.

-- robin