Weird filename encoding issue

Weird filename encoding issue

[Alexander Gladysh]

[-- Attachment #1: Type: text/plain, Size: 105 bytes --]

Hi, all!

OS X 10.5.5 (English locale), Git 1.5.5.1
Windows XP SP2 (Russian), msysGit 1.5.6.1.1071.g76fb

[-- Attachment #2: Copy.zip --]
[-- Type: application/zip, Size: 116 bytes --]

Re: Weird filename encoding issue

[Alexander Gladysh]

[-- Attachment #1: Type: text/plain, Size: 1585 bytes --]

On Fri, Oct 17, 2008 at 11:59 AM, Alexander Gladysh <agladysh@gmail.com> wrote:
> Hi, all!
>
> OS X 10.5.5 (English locale), Git 1.5.5.1
> Windows XP SP2 (Russian), msysGit 1.5.6.1.1071.g76fb

My apologies, Gmail eaten part of my message.

Sorry for such old Git versions, I can not upgrade right now. Perhaps
it was fixed in 1.6.x?

Steps to reproduce:

1. Create git repo on Windows machine, add there file, named
"Копия.txt" (Russian for Copy.txt) in CP1251 (NOT UTF-8). Just in
case, hopefully correctly named file is attached.

File must be added and committed from git gui, not from command line.

$ mkdir test && cd test
$ git init
<put file there>
$ git gui

2. Share the repo somewhere, so it would be accessible from OS X (repo
should still be on Windows machine).

3. Clone repo to OS X machine and see the weird behaviour. Git status
can't decide how file should be named, "%CA%EE%EF%E8%FF.txt" or
"\312\356\357\350\377.txt".

$ git clone /Volumes/Exchange/test/
Initialized empty Git repository in /Users/agladysh/test/.git/
$ cd test
$ git status
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	%CA%EE%EF%E8%FF.txt
nothing added to commit but untracked files present (use "git add" to track)
$ ls
%CA%EE%EF%E8%FF.txt
$ git clean -fd
Removing %CA%EE%EF%E8%FF.txt
$ git status
# On branch master
# Changed but not updated:
#   (use "git add/rm <file>..." to update what will be committed)
#
#	deleted:    "\312\356\357\350\377.txt"
#
no changes added to commit (use "git add" and/or "git commit -a")

Alexander.

[-- Attachment #2: Copy.zip --]
[-- Type: application/zip, Size: 116 bytes --]

Re: Weird filename encoding issue

[Johannes Sixt]

Alexander Gladysh schrieb:
> 3. Clone repo to OS X machine and see the weird behaviour. Git status
> can't decide how file should be named, "%CA%EE%EF%E8%FF.txt" or
> "\312\356\357\350\377.txt".

It's not 'git status' that cannot decide how to name the file. It's OS X.

> $ git status
> # On branch master
> # Untracked files:
> #   (use "git add <file>..." to include in what will be committed)
> #
> #	%CA%EE%EF%E8%FF.txt
> nothing added to commit but untracked files present (use "git add" to track)
> $ ls
> %CA%EE%EF%E8%FF.txt

git asked OS X:

   Does the file "\312\356\357\350\377.txt" exist?

and OS X said: "Yes". Because otherwise, you would have seen the file
listed as "deleted:" in the 'git status' call above.

But then git also requested a file listing from OS X in order to list the
untracked files. And now OS X returned the name "%CA%EE%EF%E8%FF.txt". How
could you expect git to tell that this is the same file when OS X cannot
decide how to name it?

The solution: Do not use file names with some local (let alone Windows
specific) encoding if you have to use the files on OS X, too.

-- Hannes

Re: Weird filename encoding issue

[Robin Rosenberg]

fredagen den 17 oktober 2008 10.55.24 skrev Johannes Sixt:
> 
> Alexander Gladysh schrieb:
> > 3. Clone repo to OS X machine and see the weird behaviour. Git status
> > can't decide how file should be named, "%CA%EE%EF%E8%FF.txt" or
> > "\312\356\357\350\377.txt".
> 
> It's not 'git status' that cannot decide how to name the file. It's OS X.
> 
> > $ git status
> > # On branch master
> > # Untracked files:
> > #   (use "git add <file>..." to include in what will be committed)
> > #
> > #	%CA%EE%EF%E8%FF.txt
> > nothing added to commit but untracked files present (use "git add" to track)
> > $ ls
> > %CA%EE%EF%E8%FF.txt
> 
> git asked OS X:
> 
>    Does the file "\312\356\357\350\377.txt" exist?
> 
> and OS X said: "Yes". Because otherwise, you would have seen the file
> listed as "deleted:" in the 'git status' call above.
> 
> But then git also requested a file listing from OS X in order to list the
> untracked files. And now OS X returned the name "%CA%EE%EF%E8%FF.txt". How
> could you expect git to tell that this is the same file when OS X cannot
> decide how to name it?
> 
> The solution: Do not use file names with some local (let alone Windows
> specific) encoding if you have to use the files on OS X, too.

That seems to be OS-X's way of saying the is a string of garbage, lets handle it as such.

Git (or msys) under windows is somewhat stupid here as it involves the eight-bit locale
despite running in a unicode OS. To get UTF-8 on XP you can set the codepage to UTF-8 
(called 65001 in windows). That might still cause problems with OS-X for some chararacters
as it wants to "decompose" the UTF-8 encoding for accented characters, which perhaps is
not a problem for you.

I have no idea how well codepage 65001 plays with msys, but it would be nice it would work.
I'm not sure how to switch codepage in msys, but if you set it from cmd and then start msys
bash from there, I'm pretty sure it will pick it up.

-- robin

Re: [msysGit] Re: Weird filename encoding issue

[Peter Krefting]

Robin Rosenberg:

> Git (or msys) under windows is somewhat stupid here as it involves
> the eight-bit locale despite running in a unicode OS. To get UTF-8 on
> XP you can set the codepage to UTF-8 (called 65001 in windows).

That presents itself with a lot of other issues, however, as the
char-based file APIs were only designed to handle up to double-byte
characters. To implement proper Unicode file name support on Windows,
one need to use the "wide" APIs (using wchar_t).

Using those APIs, it does not matter what locale the system is set to.
Everything is then UTF-16, which is trivially converted to and from
UTF-8 (and there are even native APIs to do it if one can't be bothered
to write code for it).

I have been meaning to check out a copy of the Msysgit source tree and
see how difficult it would be to replace the file API layer with proper
Unicode support, but time keeps fleeing from me, so I never seem to get
around to :-/

-- 
\\// Peter - http://www.softwolves.pp.se/

Re: Weird filename encoding issue

[Alexander Gladysh]

Thanks for answers, guys.

<...>

> git asked OS X:
>
>   Does the file "\312\356\357\350\377.txt" exist?
>
> and OS X said: "Yes". Because otherwise, you would have seen the file
> listed as "deleted:" in the 'git status' call above.
>
> But then git also requested a file listing from OS X in order to list the
> untracked files. And now OS X returned the name "%CA%EE%EF%E8%FF.txt". How
> could you expect git to tell that this is the same file when OS X cannot
> decide how to name it?
>
> The solution: Do not use file names with some local (let alone Windows
> specific) encoding if you have to use the files on OS X, too.

Thank you, but this is not the solution, this is a workaround.

1. Git hooks do not work under msysgit. Thus I'm not able to prevent
people on Windows from committing weird filenames to the repo. (Please
don't start elitist discussion that people using git must do
everything conciously.)

Note also, that even if hooks would work, such files still may appear
in the repo in theory (somebody may disable hooks after all).

2. Now I have one such file. I'm managing my repo on OS X, and have no
access to other machines right now. How can I create commit that
renames the file? Git GUI manages to create commit that adds the
renamed file, but does not delete old one.

Alexander.

Re: [msysGit] Re: Weird filename encoding issue

[Robin Rosenberg]

lördagen den 18 oktober 2008 08.22.24 skrev Alexander Gladysh:
> 2. Now I have one such file. I'm managing my repo on OS X, and have no
> access to other machines right now. How can I create commit that
> renames the file? Git GUI manages to create commit that adds the
> renamed file, but does not delete old one.

Rename the file and add it. Then git add -u to remove the old name and commit.

-- tobin

Re: Weird filename encoding issue

[Johannes Sixt]

On Samstag, 18. Oktober 2008, Alexander Gladysh wrote:
> 1. Git hooks do not work under msysgit.

This is new to me. Why are you saying that? (IOW, please be more precise with 
your description. We can't help you if all you tell us is "Does not work" .)

-- Hannes

Re: [msysGit] Re: Weird filename encoding issue

[Alexander Gladysh]

On Sat, Oct 18, 2008 at 9:51 PM, Johannes Sixt <johannes.sixt@telecom.at> wrote:
> On Samstag, 18. Oktober 2008, Alexander Gladysh wrote:
>> 1. Git hooks do not work under msysgit.
>
> This is new to me. Why are you saying that? (IOW, please be more precise with
> your description. We can't help you if all you tell us is "Does not work" .)

Err. Sorry. I thought it is a known issue:

http://groups.google.com/group/msysgit/browse_thread/thread/956e555b8d515c66

I've got an impression that msysgit 1.5.6.1.1071.g76fb we use does not
contain this fix.

Of course, I may apply it by hand on each user's machine... Still have
to do it though.

Alexander.

Re: Weird filename encoding issue

[Jakub Narebski]

"Alexander Gladysh" <agladysh@gmail.com> writes:

> Steps to reproduce:
> 
> 1. Create git repo on Windows machine, add there file, named
> "Копия.txt" (Russian for Copy.txt) in CP1251 (NOT UTF-8). Just in
> case, hopefully correctly named file is attached.
> 
> File must be added and committed from git gui, not from command line.
> 
> $ mkdir test && cd test
> $ git init
> <put file there>
> $ git gui
> 
> 2. Share the repo somewhere, so it would be accessible from OS X (repo
> should still be on Windows machine).
> 
> 3. Clone repo to OS X machine and see the weird behaviour. Git status
> can't decide how file should be named, "%CA%EE%EF%E8%FF.txt" or
> "\312\356\357\350\377.txt".

I _suspect_ that it is caused by known issue with MacOS X filesystem
in default settings.  Git assumes that what he puts in filesystem is
the same that it reads from filesystem; there is ongoing (although
bit stalled) effort to make Git work around this.  Unfortunately HFS+
does not, I think, fullfil this requirement.

-- 
Jakub Narebski
Poland
ShadeHawk on #git