Add case-challenged file system tests

Add case-challenged file system tests

[Steffen Prohaska]

There have been discussions lately on case-challenging file
systems and UTF normalization on Mac OS X.  I like to see
these problems fixed and would like to start working on a
resolution.  But I did not follow the recent discussions closely.

Is anyone actively orking on these issues?
What is the current status?

The patch below adds two simple tests that currently fail on Mac
and Windows.  I also collected Mitch's test on utf-8 and will
send it as a reply to this mail.  These two patches contain test
cases that should pass, but do currently fail.

    Steffen

-- >8 --
Git behaves strangely (from a user's point of view) on
filesystems that preserve case but do not distinguish filenames
that only differ by case.  The two major examples are Windows and
Mac OS X.  Simple operations such as "git mv" or "git merge" can
fail unexpectedly.

This commit adds two simple tests.  Both tests currently fail on
Windows and Mac, although they pass on Linux.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
---
 t/t0050-filesystems.sh |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)
 create mode 100755 t/t0050-filesystems.sh

diff --git a/t/t0050-filesystems.sh b/t/t0050-filesystems.sh
new file mode 100755
index 0000000..953b02b
--- /dev/null
+++ b/t/t0050-filesystems.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+test_description='Various filesystems issues'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+
+	touch camelcase &&
+	git add camelcase &&
+	git commit -m "initial" &&
+	git tag initial &&
+	git checkout -b topic &&
+	git mv camelcase tmp &&
+	git mv tmp CamelCase &&
+	git commit -m "rename" &&
+	git checkout -f master
+
+'
+
+test_expect_success 'rename (case change)' '
+
+	git mv camelcase CamelCase &&
+	git commit -m "rename"
+
+'
+
+test_expect_success 'merge (case change)' '
+
+	git reset --hard initial &&
+	git merge topic
+
+'
+
+
+test_done
-- 
1.5.4.40.g4a680

[PATCH] [WIP] git on MacOSX and files with decomposed utf-8 file names

[Steffen Prohaska]

From: Mitch Tishmack <mitcht.git@gmail.com>

[ spr: This patch contains Mitch's work with his original email
       as the commit message. ]

Apologies Steffen, I grabbed your CamelCase test and did a search/replace,
wasn't sure what to call it though... But I am on lunch and wanted to be
useful. Rip it apart all you want.

Fails on hfs* on OSX, works on ufs. I will bother with zfs when it can be used
again.

On UFS:
$ /bin/sh ./t0060-normalization.sh
*   ok 1: setup
*   ok 2: rename (silent normalization)
*   ok 3: merge (silent normalization)
* passed all 3 test(s)

On HFS:
$ /bin/sh t0060-normalization.sh
*   ok 1: setup
* FAIL 2: rename (silent normalization)

     git mv ä ä &&
     git commit -m "rename"

* FAIL 3: merge (silent normalization)

     git reset --hard initial &&
     git merge topic

* failed 2 among 3 test(s)

The test case, it uses perl, assuming only 5.6.1+ will work with this:

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
---
 t/t0060-normalization.sh |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)
 create mode 100755 t/t0060-normalization.sh

diff --git a/t/t0060-normalization.sh b/t/t0060-normalization.sh
new file mode 100755
index 0000000..f754fe4
--- /dev/null
+++ b/t/t0060-normalization.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+test_description='Test for silent normalization issues'
+
+. ./test-lib.sh
+
+auml=`perl -CO -e 'print pack("U",0x00E4)'`
+aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
+test_expect_success setup "
+  touch $aumlcdiar &&
+  git add $aumlcdiar &&
+  git commit -m \"initial\"
+  git tag initial &&
+  git checkout -b topic &&
+  git mv $aumlcdiar tmp &&
+  git mv tmp $auml &&
+  git commit -m \"rename\" &&
+  git checkout -f master
+
+"
+
+test_expect_success 'rename (silent normalization)' "
+
+ git mv $aumlcdiar $auml &&
+ git commit -m \"rename\"
+
+"
+
+test_expect_success 'merge (silent normalization)' '
+
+ git reset --hard initial &&
+ git merge topic
+
+'
+
+test_done
-- 
1.5.4.40.g4a680

Re: Add case-challenged file system tests

[Junio C Hamano]

Steffen Prohaska <prohaska@zib.de> writes:

> Is anyone actively orking on these issues?
> What is the current status?

Not that I know of.

> The patch below adds two simple tests that currently fail on Mac
> and Windows.  I also collected Mitch's test on utf-8 and will
> send it as a reply to this mail.  These two patches contain test
> cases that should pass, but do currently fail.

Could you fold them into one?  They are about the same topic.

Also, mark them with test_expect_failure if these are meant to
be applied.

[PATCH] Add tests for filesystem challenges (case and unicode normalization)

[Steffen Prohaska]

Unfortunately, I had no time to start the real work of fixing the
issues that are tested below.  But at least the tests should be in
good shape now and could be applied.

    Steffen

-- 8< --
Git has difficulties on file systems that do not properly
distinguish case or modify filenames in unexpected ways.  The two
major examples are Windows and Mac OS X.  Both systems preserve
case of file names but do not distinguish between filenames that
differ only by case.  Simple operations such as "git mv" or
"git merge" can fail unexpectedly.  In addition, Mac OS X normalizes
unicode, which make git's life even harder.

This commit adds tests that currently fail but should pass if
file system as decribed above are fully supported.  The test need
to be run on Windows and Mac X as they already pass on Linux.

Mitch Tishmack is the original author of the tests for unicode
normalization.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
---
 t/t0050-filesystem.sh |   67 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 67 insertions(+), 0 deletions(-)
 create mode 100755 t/t0050-filesystem.sh

diff --git a/t/t0050-filesystem.sh b/t/t0050-filesystem.sh
new file mode 100755
index 0000000..a0ab02e
--- /dev/null
+++ b/t/t0050-filesystem.sh
@@ -0,0 +1,67 @@
+#!/bin/sh
+
+test_description='Various filesystem issues'
+
+. ./test-lib.sh
+
+test_expect_success "setup case tests" '
+
+	touch camelcase &&
+	git add camelcase &&
+	git commit -m "initial" &&
+	git tag initial &&
+	git checkout -b topic &&
+	git mv camelcase tmp &&
+	git mv tmp CamelCase &&
+	git commit -m "rename" &&
+	git checkout -f master
+
+'
+
+test_expect_failure 'rename (case change)' '
+
+	git mv camelcase CamelCase &&
+	git commit -m "rename"
+
+'
+
+test_expect_failure 'merge (case change)' '
+
+	git reset --hard initial &&
+	git merge topic
+
+'
+
+auml=`perl -CO -e 'print pack("U",0x00E4)'`
+aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
+test_expect_success "setup unicode normalization tests" "
+
+  test_create_repo unicode &&
+  cd unicode &&
+  touch $aumlcdiar &&
+  git add $aumlcdiar &&
+  git commit -m \"initial\"
+  git tag initial &&
+  git checkout -b topic &&
+  git mv $aumlcdiar tmp &&
+  git mv tmp $auml &&
+  git commit -m \"rename\" &&
+  git checkout -f master
+
+"
+
+test_expect_failure 'rename (silent unicode normalization)' "
+
+ git mv $aumlcdiar $auml &&
+ git commit -m \"rename\"
+
+"
+
+test_expect_failure 'merge (silent unicode normalization)' '
+
+ git reset --hard initial &&
+ git merge topic
+
+'
+
+test_done
-- 
1.5.4.3.310.g78af

Re: [PATCH] Add tests for filesystem challenges (case and unicode normalization)

[Junio C Hamano]

Steffen Prohaska <prohaska@zib.de> writes:

> Unfortunately, I had no time to start the real work of fixing the
> issues that are tested below.

That's fine.  We are not in a hurry.

> But at least the tests should be in
> good shape now and could be applied.

I do not think we would want to see "FIXED" on systems that
already behave sanely, so we would want a fix-up like this on
top of your patch, and it would be a good to go.

I do not have a handy way to test this, though, so can you try
it out and make sure test_case and test_unicode are set to
test_expect_failure correctly on problematic filesystems?

Thanks.

--

 t/t0050-filesystem.sh |   38 ++++++++++++++++++++++++++++++++------
 1 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/t/t0050-filesystem.sh b/t/t0050-filesystem.sh
index a0ab02e..b395c22 100755
--- a/t/t0050-filesystem.sh
+++ b/t/t0050-filesystem.sh
@@ -4,6 +4,34 @@ test_description='Various filesystem issues'
 
 . ./test-lib.sh
 
+auml=`perl -CO -e 'print pack("U",0x00E4)'`
+aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
+
+test_expect_success 'see if we expect ' '
+
+	test_case=test_expect_success
+	test_unicode=test_expect_success
+	mkdir junk &&
+	echo good >junk/CamelCase &&
+	echo bad >junk/camelcase &&
+	if test "$(cat junk/CamelCase)" != good
+	then
+		test_camel=test_expect_failure
+		say "will test on a case insensitive filesystem"
+	fi &&
+	rm -fr junk &&
+	mkdir junk &&
+	>junk/"$auml" &&
+	case "$(cd junk && echo *)" in
+	"$aumlcdiar")
+		test_unicode=test_expect_failure
+		say "will test on a unicode corrupting filesystem"
+		;;
+	*)	;;
+	esac &&
+	rm -fr junk
+'
+
 test_expect_success "setup case tests" '
 
 	touch camelcase &&
@@ -18,22 +46,20 @@ test_expect_success "setup case tests" '
 
 '
 
-test_expect_failure 'rename (case change)' '
+$test_case 'rename (case change)' '
 
 	git mv camelcase CamelCase &&
 	git commit -m "rename"
 
 '
 
-test_expect_failure 'merge (case change)' '
+$test_case 'merge (case change)' '
 
 	git reset --hard initial &&
 	git merge topic
 
 '
 
-auml=`perl -CO -e 'print pack("U",0x00E4)'`
-aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
 test_expect_success "setup unicode normalization tests" "
 
   test_create_repo unicode &&
@@ -50,14 +76,14 @@ test_expect_success "setup unicode normalization tests" "
 
 "
 
-test_expect_failure 'rename (silent unicode normalization)' "
+$test_unicode 'rename (silent unicode normalization)' "
 
  git mv $aumlcdiar $auml &&
  git commit -m \"rename\"
 
 "
 
-test_expect_failure 'merge (silent unicode normalization)' '
+$test_unicode 'merge (silent unicode normalization)' '
 
  git reset --hard initial &&
  git merge topic

Re: [PATCH] Add tests for filesystem challenges (case and unicode normalization)

[Steffen Prohaska]

On Feb 25, 2008, at 9:44 PM, Junio C Hamano wrote:

> Steffen Prohaska <prohaska@zib.de> writes:
>
>> Unfortunately, I had no time to start the real work of fixing the
>> issues that are tested below.
>
> That's fine.  We are not in a hurry.

Well, I am, ... kind of.  git constantly stumbles over a filename
whose case changed.  But fortunately I do not need to wait much
longer until history will eventually have grown sufficiently such
that the rename will no longer surface ;-)


>> But at least the tests should be in
>> good shape now and could be applied.
>
> I do not think we would want to see "FIXED" on systems that
> already behave sanely, so we would want a fix-up like this on
> top of your patch, and it would be a good to go.
>
> I do not have a handy way to test this, though, so can you try
> it out and make sure test_case and test_unicode are set to
> test_expect_failure correctly on problematic filesystems?

This is a sensible way of handling this.  It works for me,
except for ...


> --
>
>  t/t0050-filesystem.sh |   38 ++++++++++++++++++++++++++++++++------
>  1 files changed, 32 insertions(+), 6 deletions(-)
>
> diff --git a/t/t0050-filesystem.sh b/t/t0050-filesystem.sh
> index a0ab02e..b395c22 100755
> --- a/t/t0050-filesystem.sh
> +++ b/t/t0050-filesystem.sh
> @@ -4,6 +4,34 @@ test_description='Various filesystem issues'
>
>  . ./test-lib.sh
>
> +auml=`perl -CO -e 'print pack("U",0x00E4)'`
> +aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
> +
> +test_expect_success 'see if we expect ' '
> +
> +	test_case=test_expect_success
> +	test_unicode=test_expect_success
> +	mkdir junk &&
> +	echo good >junk/CamelCase &&
> +	echo bad >junk/camelcase &&
> +	if test "$(cat junk/CamelCase)" != good
> +	then
> +		test_camel=test_expect_failure

test_case (instead of test_camel).

		Steffen

Re: [PATCH] Add tests for filesystem challenges (case and unicode normalization)

[Junio C Hamano]

Steffen Prohaska <prohaska@zib.de> writes:

> This is a sensible way of handling this.  It works for me,
> except for ...
> ...
> test_case (instead of test_camel).

Thanks for testing.  Fixed.