EOL handling (EGit/svn/Windows)

EOL handling (EGit/svn/Windows)

[Piotr Krukowiecki]

Hi,

in short: how to do svn->git conversion, or how to configure git/EGit
so that Windows/Linux users are happy with EOLs?


Long story:

recently I converted (with git-svn) an svn repository to git. This
created git repository with Windows (CRLF) end of line files. I
suppose the original svn repository had Windows EOLs (I did the
conversion on linux). The development is done in Eclipse (mostly on
Windows), so the EGit is a natural choice for git client.

But the problem appears: what should be the core.eol and core.autocrlf settings?

EGit seems to have problems with core.autocrlf=true - it always
converts files to LF on commit. So if someone modifies one line in a
file which was not modified yet since the git->svn conversion, each
line is seen as modified (CRLF changed to LF). It seems like a bug in
EGit - git-config man page says "files that contain CRLF in the
repository will not be touched" for core.autocrlf=true. "Normal"
command line git works better - if a file has CRLF in it in
repository, it will not convert it (it will convert new files from
CRLF to LF though - that's OK).

I could also set core.autocrlf=false which should avoid any conversion
to LF on check-in. This will move the responsibility of using correct
EOL to users. Everyone, both Windows and Linux users, will have to use
the same EOLs. Probably core.eol will have to be set to crlf, the same
as in repo. But I suppose this will not work - new files will be
created with native EOLs so the files will end up being checked in
like that. I also think it might create some problems with programs
not accepting non-native EOLs. On the other hand, it would prevent
problems caused by CRLF->LF conversion on "binary" files incorrectly
detected as "text" files...

So maybe I should re-write the repository (all the history) so that
the files are converted to LF? (examples:
http://superuser.com/questions/293941/rewrite-git-history-to-replace-all-crlf-to-lf
http://blog.gyoshev.net/2013/08/normalizing-line-endings-in-git-repositories/).This
way I should be able to use core.autocrlf for both git and EGit.
That's what I'm favoring currently. The downside is that, if I assume
correctly that SVN repo kept files in CRLF format, the conversion to
LF might break something (not sure what - some UTF files? or some
unexpected stuff...).

This re-writing of repo would not be needed if EGit did work like
normal git... should I drop EGit? I would prefer not to :(

Any other suggestions?

BTW, core.autocrlf documentation in git-config is a bit hard to
understand... it's not obvious what it affects - working dir? commits?
Does it convert to CRLF or LF? What is the "input" setting (what is
"output conversion")?  I found some explanation here:
http://thread.gmane.org/gmane.comp.version-control.git/174413


Thanks,
-- 
Piotr Krukowiecki

Re: EOL handling (EGit/svn/Windows)

[Torsten Bögershausen]

On 02/09/2015 11:22 PM, Piotr Krukowiecki wrote:
 >Any other suggestions?

My, somewhat personally, suggestion:
If there is more than one developer, don't use core.autocrlf at all-
it is a local setting, which doesn't travel through the repo, and
is slightly different in Git Egit, depending on the version.

Git for Windows has it enabled by default, Cygwin has not.

Which Git versions are you using ?
How many people are there involved, how many on Windows, how many on Linux ?
Do you want to commit to svn, or is this a one-time conversion ?

If it is a one-time conversion, and you continue to work in Git only,
then the cleanest, most portable and future proof way is to use the 
.gitattributes file,
add that to the repo, do the normalization  and push.
A line like this:
* text=auto
is the easiest way.

Have a look at
https://www.kernel.org/pub/software/scm/git/docs/gitattributes.html
take a tee or coffee, do some experiments first with a dummy repo,
but all the client OS/Gits involved.
Please let us know the result (or feel free to ask more questions)

Re: EOL handling (EGit/svn/Windows)

[Piotr Krukowiecki]

On Tue, Feb 10, 2015 at 6:49 AM, Torsten Bögershausen <tboegi@web.de> wrote:
> Which Git versions are you using ?

The one I'm testing currently:

git version 1.7.9 (cygwin)
git version 1.9.0.msysgit.0 (msys)
EGit from Eclipse Luna

Cygwin git is a bit old, as I see now. Will try to update later.


> How many people are there involved, how many on Windows, how many on Linux ?

Less than 10 actively, most on Windows.


> Do you want to commit to svn, or is this a one-time conversion ?

One-time.


> If it is a one-time conversion, and you continue to work in Git only,
> then the cleanest, most portable and future proof way is to use the
> .gitattributes file,

I'm not sure if EGit supports .gitattributes:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=342372


> add that to the repo, do the normalization  and push.
> A line like this:
> * text=auto
> is the easiest way.

I'm trying it.


> Have a look at
> https://www.kernel.org/pub/software/scm/git/docs/gitattributes.html
> take a tee or coffee, do some experiments first with a dummy repo,
> but all the client OS/Gits involved.

That's one step we did not do carefully enough :(


> Please let us know the result (or feel free to ask more questions)

For testing, I've converted all files to LF and commited it, also
added the .gitattribute file.

So far:
1. msysgit can't checkout a one file (saying filename too long, the
relative path has 215 bytes) - probably not related to EOL issue.
Cygwin git works ok. So I did not check how msysgit works yet.
2. maybe due to old cygwin git, I have a problem of not displaying
changes, if the changed line has LF eol (and the file was checked out
on Windows with CRLF eols). Will try later with newer git.
2a. EGit handles such files gracefuly, but OTOH if the file is simple
dos2unix'ed, it shows diffs showing all lines changed, and when you
commit the files, it will create empty commit.


$ git status
# On branch master
#
nothing to commit (working directory clean)

$ file master/settings.gradle
master/settings.gradle: ASCII text, with CRLF line terminators

$ dos2unix.exe master/settings.gradle
dos2unix: converting file master/settings.gradle to Unix format ...

$ git status
# On branch master
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   master/settings.gradle
#
no changes added to commit (use "git add" and/or "git commit -a")

$ git diff
fatal: LF would be replaced by CRLF in master/settings.gradle

$ echo "hi" >> master/settings.gradle

$ file master/settings.gradle
master/settings.gradle: ASCII text

############### diff does not show changes! ###################
$ git diff
fatal: LF would be replaced by CRLF in master/settings.gradle

$ git diff -- master/settings.gradle
fatal: LF would be replaced by CRLF in master/settings.gradle

$ cat master/settings.gradle
[the changes are there]

$ unix2dos.exe master/settings.gradle
unix2dos: converting file master/settings.gradle to DOS format ...

$ git diff
diff --git a/master/settings.gradle b/master/settings.gradle
index a8d6609..7aa9e6b 100755
[changes are shown]

$ vim -b master/settings.gradle
[remove CR from the changed line]

$ git status
# On branch master
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   master/settings.gradle
#
no changes added to commit (use "git add" and/or "git commit -a")

$ git diff
fatal: LF would be replaced by CRLF in master/settings.gradle



-- 
Piotr Krukowiecki

Re: EOL handling (EGit/svn/Windows)

[Torsten Bögershausen]

On 2015-02-10 11.52, Piotr Krukowiecki wrote:
> On Tue, Feb 10, 2015 at 6:49 AM, Torsten Bögershausen <tboegi@web.de> wrote:
>> Which Git versions are you using ?
> 
> The one I'm testing currently:
> 
> git version 1.7.9 (cygwin)
> git version 1.9.0.msysgit.0 (msys)
> EGit from Eclipse Luna
> 
> Cygwin git is a bit old, as I see now. Will try to update later.
> 
> 
>> How many people are there involved, how many on Windows, how many on Linux ?
> 
> Less than 10 actively, most on Windows.
> 
> 
>> Do you want to commit to svn, or is this a one-time conversion ?
> 
> One-time.
> 
> 
>> If it is a one-time conversion, and you continue to work in Git only,
>> then the cleanest, most portable and future proof way is to use the
>> .gitattributes file,
> 
> I'm not sure if EGit supports .gitattributes:
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=342372
> 
> 
>> add that to the repo, do the normalization  and push.
>> A line like this:
>> * text=auto
>> is the easiest way.
> 
> I'm trying it.
> 
> 
>> Have a look at
>> https://www.kernel.org/pub/software/scm/git/docs/gitattributes.html
>> take a tee or coffee, do some experiments first with a dummy repo,
>> but all the client OS/Gits involved.
> 
> That's one step we did not do carefully enough :(
> 
> 
>> Please let us know the result (or feel free to ask more questions)
> 
> For testing, I've converted all files to LF and commited it, also
> added the .gitattribute file.
> 
> So far:
> 1. msysgit can't checkout a one file (saying filename too long, the
> relative path has 215 bytes) - probably not related to EOL issue.
Please have a look here:
https://github.com/msysgit/msysgit/releases/Git-1.9.5-preview20141217
I think we have support for long path names (Haven't tested it myself)
> Cygwin git works ok. So I did not check how msysgit works yet.
> 2. maybe due to old cygwin git, I have a problem of not displaying
> changes, if the changed line has LF eol (and the file was checked out
> on Windows with CRLF eols). Will try later with newer git.
Normally you will not see any changes, and "git diff" will not show
anything either.
> 2a. EGit handles such files gracefuly, but OTOH if the file is simple
> dos2unix'ed, it shows diffs showing all lines changed, and when you
> commit the files, it will create empty commit.
Why this dos2unix ?
Is there a special reason ?
By the way, when people only use Egit, I assume they use Eclipse,
and you don't use Notepad.exe or so at all.
Then you don't need CRLF in the worktree at all, as Eclipse handle
LF well.

and in this case you should be able to set
git config core.autocrlf input
on all repos, just in case someone sneaks in a CRLF somewhere.
(And after the normalizing of course)

https://www.kernel.org/pub/software/scm/git/docs/git-config.html

(and don't ask me if Egit supports that)

> 
> 
> $ git status
> # On branch master
> #
> nothing to commit (working directory clean)
> 
> $ file master/settings.gradle
> master/settings.gradle: ASCII text, with CRLF line terminators
That is under msysgit ?
(Side note: Msysgit is called Git for Windows these days)
> 
> $ dos2unix.exe master/settings.gradle
Is this under Msysgit ?
> dos2unix: converting file master/settings.gradle to Unix format ...
> 
> $ git status
> # On branch master
> #
> # Changes not staged for commit:
> #   (use "git add <file>..." to update what will be committed)
> #   (use "git checkout -- <file>..." to discard changes in working directory)
> #
> #       modified:   master/settings.gradle
> #
> no changes added to commit (use "git add" and/or "git commit -a")
> 
> $ git diff
> fatal: LF would be replaced by CRLF in master/settings.gradle
That's interesting.

What does 
git config -l | grep core
give ?

Re: EOL handling (EGit/svn/Windows)

[Piotr Krukowiecki]

On Tue, Feb 10, 2015 at 4:32 PM, Torsten Bögershausen <tboegi@web.de> wrote:
> On 2015-02-10 11.52, Piotr Krukowiecki wrote:
>>
>> So far:
>> 1. msysgit can't checkout a one file (saying filename too long, the
>> relative path has 215 bytes) - probably not related to EOL issue.
>> Cygwin git works ok. So I did not check how msysgit works yet.
>
> Please have a look here:
> https://github.com/msysgit/msysgit/releases/Git-1.9.5-preview20141217
> I think we have support for long path names (Haven't tested it myself)

Thanks, I'll have a look later if I have some free time. Since
Cygwin's git is more recent I'd prefer to use it instead of "msys",
unless there are some other advantages of msys version.


>> 2. maybe due to old cygwin git, I have a problem of not displaying
>> changes, if the changed line has LF eol (and the file was checked out
>> on Windows with CRLF eols). Will try later with newer git.
> Normally you will not see any changes, and "git diff" will not show
> anything either.

I meant that there are some actual changes in the file (not only
whitespaces), but the line with the changes also has LF eol instead of
CRLF, and the actual changes are now shown. That's what the examples
below show.


>> 2a. EGit handles such files gracefuly, but OTOH if the file is simple
>> dos2unix'ed, it shows diffs showing all lines changed, and when you
>> commit the files, it will create empty commit.
> Why this dos2unix ?
> Is there a special reason ?

Just an use case - if for some reason someone/something coverts the
file to LF eol. Plus I think it would be better if empty commit was
not done (since there are no actual changes besides LF<->CRLF).


> By the way, when people only use Egit, I assume they use Eclipse,
> and you don't use Notepad.exe or so at all.
> Then you don't need CRLF in the worktree at all, as Eclipse handle
> LF well.

That's true, but I thought it'd be better to use native EOLs. The reasons:
- new files will have initially CRLF (and will be converted to LF on
first commit, but I think they will be left with CRLF in workspace
even after commit), so workspace might have CRLF files anyway
- some files are c# code/projects and are developer in VS, so they
should probably have CRLF eols; not checked how VS will work yet.
- some external tools might not work with non-native EOLs (not that I
know of any specific tool we use - just a precaution)
- when in Rome, do as Romans do, generally ;)


> and in this case you should be able to set
> git config core.autocrlf input
> on all repos, just in case someone sneaks in a CRLF somewhere.
> (And after the normalizing of course)

I'm aware of that option, we might change to it - but I still don't
see what's the advantage, except lack of CRLF<->LF conversion (that's
a valid reason, but won't core.safecrlf help here?)


>> $ file master/settings.gradle
>> master/settings.gradle: ASCII text, with CRLF line terminators
> That is under msysgit ?
> (Side note: Msysgit is called Git for Windows these days)

Nope - both repository and client was cygwin git.


>> $ dos2unix.exe master/settings.gradle
> Is this under Msysgit ?

Cygwin.


>> dos2unix: converting file master/settings.gradle to Unix format ...
>>
>> $ git status
>> # On branch master
>> #
>> # Changes not staged for commit:
>> #   (use "git add <file>..." to update what will be committed)
>> #   (use "git checkout -- <file>..." to discard changes in working directory)
>> #
>> #       modified:   master/settings.gradle
>> #
>> no changes added to commit (use "git add" and/or "git commit -a")
>>
>> $ git diff
>> fatal: LF would be replaced by CRLF in master/settings.gradle
> That's interesting.
>
> What does
> git config -l | grep core
> give ?

core.autocrlf=true
core.safecrlf=true
core.eol=native
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true

Plus, "*.gradle text" in .gitattributes.

As I wrote, it might be old Cygwin git - 1.7.9. Will try to update.


-- 
Piotr Krukowiecki

Re: EOL handling (EGit/svn/Windows)

[Piotr Krukowiecki]

On Tue, Feb 10, 2015 at 6:49 AM, Torsten Bögershausen <tboegi@web.de> wrote:
> add that to the repo, do the normalization  and push.

I did LF normalization over history. Looks like everything worked
correctly, just wondering, what does the WARNING mean?

$ git filter-branch --tree-filter '~/fix-eol.sh' -d /dev/shm/repo-tmp -- --all
Rewrite 4eab51b65af61d2b0e56f9990cc14be1e1372194 (13349/13349)
Ref 'refs/heads/master' was rewritten
Ref 'refs/remotes/origin/master' was rewritten
WARNING: Ref 'refs/remotes/origin/master' is unchanged

$ cat ~/fix-eol.sh
find . -type f \! -path ./.git/\* -print0 | xargs -0 dos2unix
--keepdate 2>/dev/null


-- 
Piotr Krukowiecki