Re: Re: Moving a directory into another fails

[Shawn Pearce <spearce@spearce.org>]

Linus Torvalds <torvalds@osdl.org> wrote:
> You guys are ignoring the _real_ problem. 
> 
> It has nothing at all to do with dependencies on external packages. The 
> REAL problem is that if you do locale-dependent trees and other git 
> objects, git will STOP WORKING.

Yes!

In jgit I assumed all tree entry names were encoded in UTF8.
Then I later learned they aren't.  Foolish me.

As Linus points out its a HUGE problem that the caller of
git-write-tree gets to decide what encoding should be used for
that tree.  Especially if someone else wants to use a different
encoding for the same filename (think ISO-8859-1 vs. UTF-8)!

I'd rather just force the tree entry names to be encoded in UTF-8
always, as its compact for most western texts (which many filenames
are), and at least degrades to supporting the non western texts.

A per-project setting is essentially impossible as we have
no such concept today, and a per-repository setting (like
i18n.commitEncoding) lets two different users encode the same
filename differently, which means two different tree SHA1s with
the exact same content... not correct!
 
> This is true for all levels of the git archive. It's true for blob 
> content, it's true for filenames in trees, and it is true for commits. The 
> commit message is actually somewhat easier (because we have nothing to 
> "compare" it to afterwards in the checked-out tree), so the commit message 
> is the _one_ thing we can kind of play games with, but even there, once 
> it's done, it's done, and it's just a stream of bytes.

Commit encoding is a problem.  Clearly the "header parts"
(tree, parent) are US-ASCII but the author and committer lines
can be anything.  So can the body.  And we have no way of knowing
what encoding was used years later, we can only guess and display
it wrong.

We really should either normalize all commit messages to a single
encoding (again, UTF-8) or embed the encoding as part of the headers
somehow (e.g. look at how XML embeds the document encoding in the
start of the document).

--