git rebase fail with CRLF conversion

git rebase fail with CRLF conversion

[Yann Droneaud]

Hi (again),

Following my previous email "Tracking vendor release with Git" [1][2],
and the advice from Git users/developers, I'm trying to use 
.gitattributes
to handle CRLF/LF conversion.

I'm following advices from:

  - "Dealing with line endings"
    https://help.github.com/articles/dealing-with-line-endings

  - "Tim's Blog, Mind the End of Your Line"
    http://timclem.wordpress.com/2012/03/01/mind-the-end-of-your-line/

  - "7.2 Customizing Git - Git Attributes"
    http://git-scm.com/book/ch7-2.html

  - "gitattributes(5) Manual Page"
    https://www.kernel.org/pub/software/scm/git/docs/gitattributes.html

While testing the behavor of Git regarding CRLF handling,
I've found git rebase fail when trying to apply a commit
that's going to fix/convert CRLF to LF.

I'm using the following script:

A file with CRLF end of line (EOL) "test" is added and commited,
then a file .gitattributes is added and commited.
The .gitattributes has a single line "test text" to ask git to do
EOL conversion.
After adding this .gitattributes file, Git start complaining about
the line ending to file "test". The file is fixed and commited.

Then this branch is rebased on top of another parent.
And here come the failures:


Most of the time git rebase failed with the following error:

  ...
  First, rewinding head to replay your work on top of it...
  Applying: Commit raw
  Applying: Added git attributes
  Applying: Normalization
  Using index info to reconstruct a base tree...
  Falling back to patching base and 3-way merge...
  error: Your local changes to the following files would be overwritten 
by merge:
        test
  Please, commit your changes or stash them before you can merge.
  Aborting
  Failed to merge in the changes.
  Patch failed at 0003 Normalization
  ...


But sometime it failed with another error:

with core.safecrlf = warn:

  ...
  First, rewinding head to replay your work on top of it...
  Applying: Commit raw
  Applying: Added git attributes
  warning: CRLF will be replaced by LF in test.
  The file will have its original line endings in your working 
directory.
  Applying: Normalization
  fatal: unrecognized input
  Repository lacks necessary blobs to fall back on 3-way merge.
  Cannot fall back to three-way merge.
  Patch failed at 0003 Normalization
  ...

with core.safecrlf = true:

  ...
  First, rewinding head to replay your work on top of it...
  Applying: Commit raw
  Applying: Added git attributes
  fatal: CRLF would be replaced by LF in test.
  Applying: Normalization
  fatal: unrecognized input
  Repository lacks necessary blobs to fall back on 3-way merge.
  Cannot fall back to three-way merge.
  Patch failed at 0003 Normalization
  ...


In thoses two latter cases, running "git add" does not fail with a 
fatal error: it does nothing.
I need to run "touch test" to make "git add" fail with error "fatal: 
CRLF would be replaced by LF in test.


I believe "git rebase" should not fail here, but more, it must not fail 
in a different fashion randomly.

Please find in reply to this email:
  - a shell script to demonstrate the behavor
  - a patch to t0020-crlf.sh to add two rebase tests.

1. <1370970410-7935-1-git-send-email-ydroneaud@opteya.com>
2. <http://thread.gmane.org/gmane.comp.version-control.git/227466>
    <http://marc.info/?l=git&m=137097069115462&w=2>


Regards

-- 
Yann Droneaud
OPTEYA

Re: git rebase fail with CRLF conversion

[Yann Droneaud]

Hi,

Le 21.06.2013 15:41, Yann Droneaud a écrit :
>
>
> I believe "git rebase" should not fail here, but more, it must not
> fail in a different fashion randomly.
>
> Please find in reply to this email:
>  - a shell script to demonstrate the behavor


Please find a shell script to test git rebase with .gitattributes text 
flag
set to enable End-Of-Line (EOL) conversion from CRLF to LF, with 
core.safecrlf set to warn
and to true.

Regards.

-- 
Yann Droneaud
OPTEYA


----------------------------------8<----------------------------------
#! /bin/sh

set -e

LC_ALL=C
export LC_ALL

#GIT_AUTHOR_DATE=`date -R`
GIT_AUTHOR_DATE="2001-01-01 00:00:00.+00:00"
GIT_COMMITTER_DATE="$GIT_AUTHOR_DATE"

export GIT_AUTHOR_DATE
export GIT_COMMITTER_DATE

# global settings
set +e
global_core_eol=`git config --global core.eol`
global_core_safecrlf=`git config --global core.safecrlf`
global_core_autocrlf=`git config --global core.autocrlf`
set -e

# archive
if test -d work; then
     dir=`mktemp -d work.XXXXX`
     rmdir $dir
     mv work $dir
fi

mkdir work

echo "****** Testing in work *******"

cd work

# prepare
git init
git commit --allow-empty -m "empty root commit"

# local settings, might be different from the global settings
set +e
local_core_eol=`git config core.eol`
local_core_safecrlf=`git config core.safecrlf`
local_core_autocrlf=`git config core.autocrlf`
set -e

echo "global core.eol       = $global_core_eol"
echo "global core.safecrlf  = $global_core_safecrlf"
echo "global core.autocrlf  = $global_core_autocrlf"
echo "local core.eol        = $local_core_eol"
echo "local core.safecrlf   = $local_core_safecrlf"
echo "local core.autocrlf   = $local_core_autocrlf"

# set default configuration
git config --local core.eol native
git config --local core.safecrl warn
git config --local core.autocrlf false

echo "current core.eol      = `git config core.eol`"
echo "current core.safecrlf = `git config core.safecrlf`"
echo "current core.autocrlf = `git config core.autocrlf`"

CRLF="\r\n"
CR="\r"
LF="\n"

# TODO detect line ending on the current environment
if true ; then
     EOL=$LF
else
     EOL=$CRLF
fi

echo "Create work branches"
git branch import-raw master
git branch import-eol master

echo "Create a branch to be used as new root later"
git checkout import-eol
git commit --allow-empty -m "an empty commit"

file_type=

create_raw_file ()
{
     # Want to test mixed EOL:
     # printf "Hello World 1${CRLF}Hello World 2${CRLF}${CR}Hello World 
3${CRLF}Hello World 4" > test
     printf "Hello World 1${CRLF}Hello World 2${CRLF}Hello World 
3${CRLF}Hello World 4" > test
     file_type="`file test`"
     echo "::: $file_type"

}

check_file ()
{
     local t="`file test`"
     if test "x$file_type" != "x$t" ; then
	file_type="$t"
	echo "::: $file_type"
     fi
}

create_git_attributes ()
{
     printf "test text${EOL}" > .gitattributes
}

#
# First test:
#
# import raw, add .gitattributes after, normalize, rebase
#
echo "===== BEGIN: first test ====="
git checkout import-raw
git reset --hard master
create_raw_file
git add test && check_file

git commit -m "Commit raw" && check_file

create_git_attributes  && check_file
git add .gitattributes && check_file
git commit -m "Added git attributes" && check_file

echo "--- First kind of 'normalization'"
#
# trick from https://help.github.com/articles/dealing-with-line-endings
#
git rm --cached test
git reset --hard && check_file
git add test && check_file
git commit -m "Normalization"
check_file
git tag norm1

echo "--- Second kind of 'normalization'"
git reset --hard HEAD^
check_file
create_raw_file
rm test
git checkout test  && check_file
git add test && check_file
git commit -m "Normalization"
check_file
git tag norm2

echo "--- Third kind of 'normalization'"
git reset --hard HEAD^
check_file
create_raw_file
dos2unix test  && check_file
git add test  && check_file
git commit -m "Normalization"
check_file
git tag norm3

echo "--- differences ? ---"
git diff norm1 norm2
git diff norm1 norm3
git diff norm2 norm1
git diff norm2 norm3
git diff norm3 norm1
git diff norm3 norm2

echo "--- rebase, should failed ---"
git rebase import-eol || {
     echo "--- Expected failure to rebase on another branch ---"
     check_file
     git status
     git diff -w --stat
     git diff

     # just adding the file again and continue ...
     git add test && check_file
     git rebase --continue || { echo "### failed to continue rebase" ; 
exit 1 ; }
     check_file
}

echo "--- rebase complete ---"

check_file

echo "===== END: first test ====="


#
# again, with safecrlf set
#
echo "===== BEGIN: second test ====="
git config core.safecrlf true

git checkout import-raw
git reset --hard master
create_raw_file
git add test && check_file

git commit -m "Commit raw" && check_file

create_git_attributes  && check_file
git add .gitattributes && check_file
git commit -m "Added git attributes" && check_file

echo "--- First kind of 'normalization', git add should failed"
git rm --cached test
git reset --hard && check_file
git add test && { echo "### git add must failed !" ; exit 1 ; }

echo "--- Second kind of 'normalization', git add should failed"
rm test
git checkout test && check_file
git add test && { echo "### git add must failed !" ; exit 1 ; }

echo "--- Third kind of 'normalization'"
dos2unix test && check_file
git add test && check_file
git commit -m "Normalization"
check_file

echo "--- rebase, should failed ---"
git rebase import-eol || {
     echo "--- Expected failure to rebase on another branch ---"
     check_file
     git status

     set +e
     git diff -w --stat || echo "!? git diff return an error !?"
     git diff || echo "!? git diff return an error !?"
     set -e

     git add test && { echo "### git add must failed !" ; exit 1 ; }

     dos2unix test && check_file
     git add test && check_file
     git status
     git rebase --continue || { echo "### failed to continue rebase" ; 
exit 1 ; }
     check_file
}

echo "--- rebase complete ---"

check_file

echo "===== END: second test ====="

#
# the way around:
# first .gitattributes, then the file with CRLF (with safecrlf unset)
#
echo "===== BEGIN: third test ====="
git config core.safecrlf false
git checkout import-raw
git reset --hard master

create_git_attributes
git add .gitattributes
git commit -m "Added git attributes"

create_raw_file
git add test && check_file
git commit -m "Commit raw" && check_file

echo "--- First kind of 'normalization', git commit should failed"
git rm --cached test
git reset --hard && check_file
git add test && check_file
git commit -m "Normalization" && { echo "### git commit should failed 
here !" ; exit 1 ; }
check_file

echo "--- Second kind of 'normalization', git commit should failed"
rm test
git checkout test && check_file
git add test && check_file
git commit -m "Normalization" && { echo "### git commit should failed 
here !" ; exit 1 ; }
check_file

echo "--- Third kind of 'normalization', git commit should failed"
dos2unix test && check_file
git add test && check_file
git commit -m "Normalization" && { echo "### git commit should failed 
here !" ; exit 1 ; }
check_file

echo "--- rebase, should failed ---"
git rebase import-eol || {
     echo "--- Expected failure to rebase on another branch ---"
     check_file
     git status
     git diff -w --stat
     git diff

     # just adding the file again and continue ...
     git add test && check_file
     git rebase --continue || { echo "### failed to continue rebase" ; 
exit 1 ; }
     check_file
}

echo "--- rebase complete ---"

check_file

echo "===== END: third test ====="


#
# the way around
# first .gitattributes, then the file with CRLF (with safecrlf set to 
true)
#
echo "===== BEGIN: fourth test ====="
git config core.safecrlf true
git checkout import-raw
git reset --hard master

create_git_attributes
git add .gitattributes
git commit -m "Added git attributes"

create_raw_file
# with safecrlf set to true, it's impossible to add the file with CRLF.
git add test && { echo "### git add should failed here !" ; exit 1 ; }
dos2unix test && check_file
git add test && check_file
git commit -m "Commit raw" && check_file

echo "--- rebase, should failed ---"
git rebase import-eol || {
     echo "--- Expected failure to rebase on another branch ---"
     check_file
     git status
     git diff -w --stat
     git diff

     # just adding the file again and continue ...
     git add test && check_file
     git rebase --continue || { echo "### failed to continue rebase" ; 
exit 1 ; }
     check_file
}

echo "--- rebase complete ---"

check_file

echo "===== END: fourth test ====="

exit 0
----------------------------------8<----------------------------------

[PATCH] t0020-crlf: test rebase with text conversion and safecrlf

[Yann Droneaud]

This patch add two tests to reproduce the problems described
in thread "git rebase fail with CRLF conversion"
 <fb20a7d711fdd218f58f1f2090b1c515@meuh.org>
 <http://thread.gmane.org/gmane.comp.version-control.git/228613>
 <http://marc.info/?l=git&m=137182211414404&w=2>

- Add and commit a file with CRLF,
- Add and commit a .gitattributes with text flag for the file,
- Convert CRLF to LF in file, commit the change,
- Rebase the whole on top of another parent
  The rebase currently failed.

This scenario is repeated twice, first with core.safecrlf set to warn,
a second time with core.safecrlf set to true.

git diff will also fail when core.safecrlf is set to true
and the file is not already converted to LF.

Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
---
 t/t0020-crlf.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/t/t0020-crlf.sh b/t/t0020-crlf.sh
index 1a8f44c..12147ba 100755
--- a/t/t0020-crlf.sh
+++ b/t/t0020-crlf.sh
@@ -506,4 +506,56 @@ test_expect_success 'New CRLF file gets LF in repo' '
 	test_cmp alllf alllf2
 '
 
+test_expect_success 'Prepare rebase test' '
+	test_config core.autocrlf false &&
+	test_config core.safecrlf warn &&
+	test_config core.eol native &&
+	git branch rebase-initial &&
+	git checkout -b rebase-other rebase-initial &&
+	git commit --allow-empty -m "Another commit" &&
+	git checkout rebase-initial
+'
+
+# git rebase will failed to apply the "normalization" commit
+test_expect_success 'Rebase with core.safecrlf set to warn (default)' '
+	test_config core.autocrlf false &&
+	test_config core.safecrlf warn &&
+	test_config core.eol native &&
+	git checkout -b rebase-warn rebase-initial &&
+	for w in A couple of lines ; do echo $w ; done | append_cr >rebase-warn.txt &&
+	git add rebase-warn.txt && git commit -m "Added raw text" &&
+	echo rebase-warn.txt text >> .gitattributes &&
+	git add .gitattributes && git commit -m "Added .gitattributes" &&
+	git rm --cached rebase-warn.txt && git reset --hard &&
+	git add rebase-warn.txt && git commit -m "Normalized" &&
+	git rebase rebase-other || {
+		git status && git diff &&
+		git add rebase-warn.txt && git rebase --continue ;
+	}
+'
+
+#
+# git rebase will failed to apply the "normalization" commit
+# additionnaly
+# git diff and git add will failed when core.safecrlf is set to true
+#
+test_expect_success 'Rebase with core.safecrlf set to true' '
+	test_config core.autocrlf false &&
+	test_config core.safecrlf true &&
+	test_config core.eol native &&
+	git checkout -b rebase-true rebase-initial &&
+	for w in Another couple of lines ; do echo $w ; done | append_cr >rebase-true.txt &&
+	git add rebase-true.txt && git commit -m "Added raw text" &&
+	echo rebase-true.txt text >> .gitattributes &&
+	git add .gitattributes && git commit -m "Added .gitattributes" &&
+	git diff || git add rebase-true.txt ||
+	for w in Another couple of lines ; do echo $w ; done >rebase-true.txt &&
+	git add rebase-true.txt && git commit -m "Normalized" &&
+	git rebase rebase-other || {
+		git status && git diff || git add rebase-true.txt ||
+		for w in Another couple of lines ; do echo $w ; done >rebase-true.txt &&
+		git add rebase-true.txt && git rebase --continue ;
+	}
+'
+
 test_done
-- 
1.8.2.1

Re: [PATCH] t0020-crlf: test rebase with text conversion and safecrlf

[Yann Droneaud]

Le 21.06.2013 16:15, Yann Droneaud a écrit :
> This patch add two tests to reproduce the problems described
> in thread "git rebase fail with CRLF conversion"
>  <fb20a7d711fdd218f58f1f2090b1c515@meuh.org>
>  <http://thread.gmane.org/gmane.comp.version-control.git/228613>
>  <http://marc.info/?l=git&m=137182211414404&w=2>
>
> - Add and commit a file with CRLF,
> - Add and commit a .gitattributes with text flag for the file,
> - Convert CRLF to LF in file, commit the change,
> - Rebase the whole on top of another parent
>   The rebase currently failed.
>
> This scenario is repeated twice, first with core.safecrlf set to 
> warn,
> a second time with core.safecrlf set to true.
>

Note: those two tests might failed as described in message

"git rebase fail with CRLF conversion"
  <fb20a7d711fdd218f58f1f2090b1c515@meuh.org>
  <http://thread.gmane.org/gmane.comp.version-control.git/228613>
  <http://marc.info/?l=git&m=137182211414404&w=2>

I haven't found a testcase to reproduce the same exact failure at will.

So this patch should probably not used asis until the two kind of 
errors can be reproduced.

Additionnaly, It should also be tested on a CRLF system (Windows).

> git diff will also fail when core.safecrlf is set to true
> and the file is not already converted to LF.
>

It might be a test on its own.

Regards

-- 
Yann Droneaud
OPTEYA

Re: git rebase fail with CRLF conversion

[Yann Droneaud]

Le 21.06.2013 15:41, Yann Droneaud a écrit :

>
> In thoses two latter cases, running "git add" does not fail with a
> fatal error: it does nothing.
> I need to run "touch test" to make "git add" fail with error "fatal:
> CRLF would be replaced by LF in test.
>

While searching on the Internet, I've found other people complaining of 
a similar error

http://stackoverflow.com/questions/5074136/git-rebase-fails-your-local-changes-to-the-following-files-would-be-overwritte

The workaround, "git config --global core.trustctime false", might 
suggest there's a race somewhere while rebasing.

BTW, I forgot to describe my environnment:

- GNU/Linux
- Fedora 18 x86_64
- git version 1.8.1.4
- git version 1.8.3.1.448.gfb7dfaa

>
> I believe "git rebase" should not fail here, but more, it must not
> fail in a different fashion randomly.
>

Regards.

-- 
Yann Droneaud
OPTEYA