Re: [PATCH 2/2] send-email: rfc2047-quote subject lines with non-ascii characters

[Jeff King <peff@peff.net>]

On Sat, Mar 29, 2008 at 10:43:40PM +0100, Robin Rosenberg wrote:

> First that is even by random an unlikely sequence. For any "real" is string
> it simply won't happen, even in this context. Try scanning everything you
> can think of and see if you find such a sequence that is not actually UTF-8.

That's the problem I was mentioning: "everything I can think of" is
basically just us-ascii with a few accented characters. I don't know
how, e.g., Japanese texts will fare with such a test.

> > But over all commonly used encodings, what is the probability in an
> > average text of that encoding that it contains valid UTF-8?
> > For example, I have no idea what patterns can be found in EUCJP.
> 
> See here http://www.ifi.unizh.ch/mml/mduerst/papers/PDF/IUC11-UTF-8.pdf

Thanks, that is an interesting read. And he seems to indicate that you
can guess with a reasonable degree of success. But a few points on that
work:

  - he has a specific methodology for guessing, which is more elaborate
    than what you proposed. So to get his results, you would need to
    implement his method. Hopefully if perl does have a "guess if this
    looks like utf8" method, it uses a similar scheme.

  - he does admit that some encodings have difficult to assess
    probabilities, and it will vary from language to language. See page
    22:

      If a specific language does not use all three letters (a single
      letter on the left and the corresponding two letters on the
      right), then this combination presents no danger. Further checks
      can then be made with a dictionary, although there is the problem
      that a dictionary never contains all possible words, and that of
      course resource names don't necessarily have to be words.

  - he mentions Latin, Cyrillic, and Hebrew encodings. I note the
    conspicuous absence of any Asian languages.

> Note that a random string is a randomly generated string. Not a random
> string from the set of actually existing strings.

Sure. But looking at random strings isn't terribly useful; there is a
non-uniform distribution over the set of strings, dependent on the
_actual_ encoding. So there are going to be "good" encodings that will
guess well, and there will be "bad" encodings that might not (and by
"will", I mean "there may be"; that is the very thing I am saying we
don't have good evidence for).

-Peff