Re: gitweb on kernel.org and UTF-8

[Junio C Hamano <junkio@cox.net>]

Kay Sievers <kay.sievers@vrfy.org> writes:

> Should be fine now. The escapeHTML() garbled the utf8 "ö", and the
> decode() failed that.

Looking better.  Thanks.

This begs for addressing another issue, although I am hesitant
to open this can of worms at this moment.

It might be a good idea to have configuration items gitk and
gitweb can use to get a hint to decide what the commit log
message and the blob data encodings might be.  gitweb already
adds its own information to the git repository format
(.git/description), so this _could_ be stored outside just like
that (e.g. .git/commit_log_encoding), but using .git/config is
probably better.

How about doing something like this?

	[i18n]
        	commitEncoding = utf8
		blobEncoding = utf8

to mean:

	If you _have_ to make an assumption on an encoding
	commit and blob objects are in, utf8 is your best bet
	(but mistakes can happen, and some blobs can be binary).

Then gitweb and gitk can look at commitEncoding and blobEncoding
as a hint to base its display defaults on.  Sending everything
out in utf8 would be sane and safe choice these days for gitweb,
so if commitEncoding is latin-1 it may need to iconv latin-1 to
utf8 while reading commits.  For blobs, it might be better off
asking file(1) or File::MMagic (since you are using Perl in
gitweb --- sorry I do not know tcl equivalent of that) what they
are; eventually you would want to be able to show repositories
full of jpeg pictures anyway ;-).

On the commit-producing side, I could have:

	[i18n]
        	editorEncoding = latin-1

and if editorEncoding is different from commitEncoding,
"git-commit -c $commit" would first iconv from utf8 to latin-1
before populating the user's editor, and iconv back from latin-1
to utf8 before feeding what the user edited to commit-tree.

Pathname encoding is the reason why I was hesitant about bring
this up.  Although it is too late for 1.0 now, we _could_ have
declared that the paths recorded in git tree objects and index
files are internally utf8, and working tree paths can be in
different encoding.  As a local repository configuration not
project wide configuration, we could have something like:

	[i18n]
        	pathnameEncoding = latin-1

to mean that the filesystem paths returned by readdir(3) and
accepted by open(2) and friends are in latin-1.  Comparison and
movement between working tree files, the index file, and tree
objects have to involve iconv and do the right thing.  So if you
fetch from such a repository into a filesystem that stores
pathnames in utf8, the right thing should happen.

I personally feel any sane project should restrict its pathname
to ASCII only, so this issue might be moot (or is the right word
"mute"?), but something like this _might_ be useful in later
versions of git.

But not in 1.0.