Re: GIT character codecs

Re: GIT character codecs

[Marco Costalba]

Matthias Urlichs wrote:

>Hi, Junio C Hamano wrote:
>
>>I suspect you could just take the default from LANG and LC_CTYPE
>>to cover 98% of the cases, and to cover the rest 2% the
>>problematic cases have a per-window override menu.
>
>
>Or, if you want to be super-flexible, just try to decode as UTF-8.
>Success? fine -- otherwise use whatever your application's default is
>set to, or your local encoding (which may of course also be UTF-8), or
>fall back to 8859-15.
>

How it is possible to test for success?

The codec could see everything decodable, but It's only from the context,
i.e. the user who reads, that it is possible to realize it's the wrong codec.  

Anyhow, as I posted previous in this thread, qgit-0.97.2 now has a codec selector
 as a persistent setting, defaulting to local codec.


  Marco



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: GIT character codecs

[Matthias Urlichs]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

Hi,

Marco Costalba:
> >Or, if you want to be super-flexible, just try to decode as UTF-8.
> >Success? fine -- otherwise use whatever your application's default is
> >set to, or your local encoding (which may of course also be UTF-8), or
> >fall back to 8859-15.
> 
> How it is possible to test for success?

UTF-8 decoding does NOT always succeed.

In fact, there is no other 8-bit encoding which decodes as valid UTF-8,
given real-world text.

> The codec could see everything decodable, but It's only from the
> context, i.e. the user who reads, that it is possible to realize it's
> the wrong codec.  
> 
True in general, but not with UTF-8 (or ASCII, for that matter ;-).

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
You can fool the people about many things, but only a fool would be foolish
enough to fool the people about money.
					-- Italo Bombolini

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: GIT character codecs

[Marco Costalba]

Linus Torvalds wrote:

>
>On Sun, 13 Nov 2005, Marco Costalba wrote:
>
>>If encoding is a per-blob _and_ per-log message property a real solution, although cumbersone,
>>could be that git stores encoding togheter with the blob and the commits.
>
>
>We'd be much better off with just saying "we encourage people to use 
>utf-8, but if you don't, just set your locale to make things show up 
>properly".
>
>utf-8 is clearly the future, and if we make git internally aware of 
>locales, that's just going to complicate things. And usually for no good 
>reason, since users don't really care that much.
>

Yes, I agree with you and Junio, a workaround like let the user to choose the code page 
in the viewer is more then enough.

And also in this case, a choice between the local codec and utf-8 should take almost 
all cases.

Because of this, I added a text codec selector in qgit, reachable from settings menu.

There is a new release, qgit-0.97.2, with this new feature and some good
annotate fixes.

For the people interested the page with all the links is:
 http://digilander.libero.it/mcostalba/

  Marco

P.S: Now I can finally see Junio japanese name ;-)



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: GIT character codecs

[Marco Costalba]

Junio C Hamano wrote:

>Marco Costalba <mcostalba@yahoo.it> writes:
>
>>I modified qgit to force the use of utf-8 codec instead of the local one, 
>>in my case ISO-8859-15.
>
>
>I suspect that trying to have a globa single encoding is a wrong
>approach overall.  There are a handful issues to think about.
>

I agree.

>First the easiest one -- the commit log messages.  We encourage
>use of UTF-8, 
>
> [cut]
>
>But we do not _enforce_ UTF-8.
>
>What this means for qgit is that it is often sufficient for its
>log browser to support one encoding at a time, provided if it
>allows the user to switch which encoding to use depending on
>what project is being viewed.
>
>
> [cut]
>
>
>But other projects may use different
>encodings, and even a single project can have its i18n message
>files in different encodings and charsets in different files.
>Users probably want to be able to view all of them, even if they
>only understand a couple of languages and not others.
>
>What this means for qgit is that at least you should be able to
>show a whole file in a single encoding, but if you show more
>than two files at the same time, one in each window, these
>windows may be showing its contents in different encodings and
>charsets.  So you would need to give a way to your users to tell
>you what encoding each file is in.  Using global locale as the
>default and having a way to override that per file basis would
>be sufficient.
>

If encoding is a per-blob _and_ per-log message property a real solution, although cumbersone,
could be that git stores encoding togheter with the blob and the commits.

The interfaces to read out encoded info could be something like 

git-rev-list --pretty  --encoding
git-ls-files -e

for commits and blobs respectively.

This avoids all user settings. More, the user could do not know the encoding, as
example when browsing a public repo. 


But I understand is too late for something like this. So what qgit can do is
to show a setting with a list of all supported encodings from the user to choose from.

But this is also a problem because, first we should have _two_ settings, one from log 
messages and one from blobs _and_ also in this case we miss the problem of different 
blobs with different encoding.

A practical workaround, *that do not catches all cases at all* could be qgit has only one global
setting (defaulting to local codec) and applies that setting blindly to everything, from commits
to blobs, the user changes that setting and refreshes the view until he finds the correct one.

But I'm a bit reclutant to add this 'codec list' stuff because, as said, is *not* the real
solution.

   Marco




		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

Re: GIT character codecs

[Junio C Hamano]

Marco Costalba <mcostalba@yahoo.it> writes:

> But I'm a bit reclutant to add this 'codec list' stuff
> because, as said, is *not* the real solution.

Why isn't it real?  UNIX files do not declare what encoding they
are in, and people get their work done with existing viewing or
editing tools (e.g.  vi, less, emacs, firefox [*1*], ...).  How
is your situation any different?

I suspect you could just take the default from LANG and LC_CTYPE
to cover 98% of the cases, and to cover the rest 2% the
problematic cases have a per-window override menu.  If you first
implement only the locale support without per-window override
and tell your users to set these environment variables to
iso-8859-15 when running qgit on your Italian project repository
(or use utf-8 on git.git), wouldn't that be sufficient for most
everyday uses?

[Footnotes]

*1* Ok, firefox is a bit different because they can read charset
attribute from content type, but even so they have a manual
override from View->CharEncoding menu to let their users cope
with broken pages.

Re: GIT character codecs

[Matthias Urlichs]

Hi, Junio C Hamano wrote:

> I suspect you could just take the default from LANG and LC_CTYPE
> to cover 98% of the cases, and to cover the rest 2% the
> problematic cases have a per-window override menu.

Or, if you want to be super-flexible, just try to decode as UTF-8.
Success? fine -- otherwise use whatever your application's default is
set to, or your local encoding (which may of course also be UTF-8), or
fall back to 8859-15.

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
Difficult:
	Disciplines like physics, math, engineering, which is why I don't
	want to do them.
		-- Fashionable Dictionary (www.butterfliesandwheels.com)

Re: GIT character codecs

[Linus Torvalds]

On Sun, 13 Nov 2005, Marco Costalba wrote:
> 
> If encoding is a per-blob _and_ per-log message property a real solution, although cumbersone,
> could be that git stores encoding togheter with the blob and the commits.

We'd be much better off with just saying "we encourage people to use 
utf-8, but if you don't, just set your locale to make things show up 
properly".

utf-8 is clearly the future, and if we make git internally aware of 
locales, that's just going to complicate things. And usually for no good 
reason, since users don't really care that much.

I really hate codepages. I'd much rather say:
 - git is 8-bit clean, so you can use any damn encoding you want
 - utf-8 is strongly recommended for all the same reasons it's recommended 
   for anything else.

Hmm?

		Linus

GIT character codecs

[Marco Costalba]

Hi all,

   this morning I pulled from git repo and tried to look at Junio japanese name with:

      git-diff-tree -p e6bd23911efd0a2bd756c77d9e7ba6576eb739a1

With my big sorry the name was erroneously shown on my box.

I modified qgit to force the use of utf-8 codec instead of the local one, 
in my case ISO-8859-15.

Now I can see Junio name in beautiful japanese characters, and also the 
correct 'Lukas Sandstrom' name.

The problem is now qgit on my local git archives no more correctly 
shows special italian characters.

So here we arrive at the points:

1) If utf8 is the 'only good one' for GIT repos why git-diff-tree uses 
the local codec, in my case ISO-8859-15.

2) If utf8 is not the 'only good one' but the commits are always saved using the
local codec, how is possible to set the correct codec to use after pulling 
from a remote repo with a different codec (in this case public git repo that 
is in utf8).

Thanks
Marco



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: GIT character codecs

[Junio C Hamano]

Marco Costalba <mcostalba@yahoo.it> writes:

> I modified qgit to force the use of utf-8 codec instead of the local one, 
> in my case ISO-8859-15.

I suspect that trying to have a globa single encoding is a wrong
approach overall.  There are a handful issues to think about.

First the easiest one -- the commit log messages.  We encourage
use of UTF-8, because they hold people's names which are the
significant source of i18n needs.  Even if your project starts
out with just a group of people whose names can be spelled in
ASCII just fine, or all western european in which case you might
be tempted to do latin-1 or 8859-15, or all Japanese with
EUC-JP, your project _might_ someday have a member whose name
cannot be spelled if you pick one of these encodings [*2*].  So
we encourage use of UTF-8 and help people by having -u flag in
git-mailinfo, for example.

But we do not _enforce_ UTF-8.  I've never worked actively with
non-English free software projects, and I do not know if people
in, say, Japan write their commit log messages in Japanese.  I
however think it is a reasonable thing to do if the project is
local and exclusive (e.g. a company internal project).  And if
local encoding is more convenient to handle for users
(e.g. their "less" is configured to expect something non UTF-8),
it is perfectly reasonable to do all their commit log messages
in their local encoding, as long as the members understand that
is the project's policy.  And it is probably reasonable to
assume that within one project, a single encoding is used for
commit log messages, although it may not be UTF-8.

What this means for qgit is that it is often sufficient for its
log browser to support one encoding at a time, provided if it
allows the user to switch which encoding to use depending on
what project is being viewed.

The trouble you had is however not with commit messages, but
with the blob contents (i.e. user data).  First of all, the user
data is just any binary goo for git, and the interpretation of
it is left to the Porcelains and the users.  There are two
things to talk about here.

I imagine qgit has, in addition to diff-between-revisions
viewer, a blob viewer that lets your users browse a whole file
in a given revision.  The files in the Documentation directory
in git.git/ right now happens to be encoded in UTF-8 because
they are asciidoc sources.  But other projects may use different
encodings, and even a single project can have its i18n message
files in different encodings and charsets in different files.
Users probably want to be able to view all of them, even if they
only understand a couple of languages and not others.

What this means for qgit is that at least you should be able to
show a whole file in a single encoding, but if you show more
than two files at the same time, one in each window, these
windows may be showing its contents in different encodings and
charsets.  So you would need to give a way to your users to tell
you what encoding each file is in.  Using global locale as the
default and having a way to override that per file basis would
be sufficient.

A more interesting thing about user data is the diff between
revisions.  How should you show the diff between
git-pack-redundant documentation that changes Lukas' name
encoded from latin-1 to UTF-8?

First, git-diff ("diff" it calls) is encoding agnostic.  The
lines that come out from it are taken from its input files and
if two revisions you compared both store your words in Italian
encoded in iso-8859-15, the output would be iso-8859-15.  IOW,
diff would end up doing the right thing but without knowing it.

The Lukas' name encoding patch would be handled the same way.
Diff output would just say "these lines have different bytes",
and spit out '-' line encoded in latin-1 and '+' in UTF-8.

This is a mess in one sense but not necessarily so.  If we could
say "Mr diff, the first file is in latin-1 and the second one is
in UTF-8, so compare them accordingly, and by the way we would
like the output in UTF-8 please", then it would be interesting
and may be sometimes useful, but in this particular case it is
useless -- there would not be any diff.  And the reason _I_ ran
diff before committing that change was to make sure I did the
encoding conversion correctly, so noticing the byte-level
differences was the right thing to do.

What should qgit do about this?  This "encoding gotcha fix
patch" is a special case, so I think it is perfectly reasonable
if qgit did not do anything special about it.  So the "per-file
encoding override" I suggested for whole-file viewer would be
adequate.  For the special "encoding fix" case, as I discussed
above, the most useful output is "raw bytes", so maybe throw in
a special encoding "raw" as one of the choices and I think you
are done.


[Footnote]

*1* Some people may argue unicode cannot spell all names, but
let's pretend it can -- or at least it can cover wider character
sets than any single local encodings.  Some people further may
argue that iso-2022 does better, but let's not go there.

*2* We do not forbid you to store arbitrary binary blob in your
commit objects.