Re: OT: character encodings (was: Linux 2.6.20-rc4)

[Willy Tarreau <w@1wt.eu>]

On Sun, Jan 07, 2007 at 08:11:38PM +0100, Jan Engelhardt wrote:
> 
> On Jan 7 2007 17:06, Russell King wrote:
> >On Mon, Jan 08, 2007 at 12:29:05AM +0800, David Woodhouse wrote:
> >
> >$ git log | head -n 1000 | tail -n 200 > o
> >$ file -i o
> >o: text/plain; charset=us-ascii
> >$ git log | head -n 1000 | tail -n 300 > o
> >$ file -i o
> >o: text/plain; charset=us-ascii
> >$ git log | head -n 1000 | tail -n 400 > o
> >$ file -i o
> >o: text/plain; charset=utf-8
> 
> I am inclined to say that "file" does not count, because it tries to guess an
> ambiguous mapping from bytes to character set. Even more, file should be
> _unable at all_ to distinguish an iso-8859-1 from an iso-8859-2 (or worse: 15)
> file. This program is soo... forget it, it's not an argument. It works well for
> headerful files, but text files don't really contain one. The next best thing
> would be html, with a proper <meta http-equiv=Content> tag.

The stupidity from the start up with those character sets is that they
consider that a whole file is written with a given set. In fact, the
charset should apply to characters themselves. At least, the
quoted-printable, non-human friendly, encoding was the least stupid.

Now that UTF8 comes everywhere, everyone receives tons of mangled mails,
and even mailers which correctly support UTF8 and use it by default manage
to shoot themselves in the foot when they reply to, or forward a mail. The
system is completely broken because limited by design, and we have to learn
to live with this brokenness.

Willy