UTF-8 trouble with GIT 1.5.0.3

[Willy Tarreau <w@1wt.eu>]

Hi Junio,

after upgrading from GIT 1.4 to 1.5, I noticed that I had got
errors when applying patches from mailboxes and that I had to 
manually add "--no-utf8" to git-am. The error was always :

  "fatal: cannot convert from latin1 to utf-8".

This morning, I decided to take a closer look at it, first
trying to fix it in the configuration. Even this fails :

[i18n]
        commitencoding = latin1

$ git-am -3 username.patch 
fatal: cannot convert from latin1 to latin1


So I took a look at the code and noticed that in builtin-mailinfo,
convert_to_utf8() is always called because metainfo_charset is
always set. Also, this function always calls reencode_string(),
no matter what input and outputs are passed. And since it fails
for an unknown reason, I get the error message.

Now I have several options :
 
 - always pass --no-utf8 with git-am to leave metainfo_charset
   NULL and avoid the call to convert_to_utf8()
   => very annoying and I have to rm -rf .dotest every time I forget
      about it

 - add a check to convert_to_utf8() so that when input and output
   charsets are equal, it does not malloc/convert/copy/free the
   result. This would seem necessary anyway.

 - check for empty charset name instead of just a NULL pointer,
   so that it is still possible by configuration to stay compatible
   with old default behaviour which is : not to convert by default.

 - switch back to 1.4 : this will be my immediate option while
   waiting for any advice on the right long-term solution.

I would be glad to get a recommendation for one of the solutions
above. I can workout a patch, but I don't want to waste my time
doing crappy patches that will not be retained.

Please note that I don't know a damn thing about the way iconv
works, which may be one of the reason I really hate it. So it
will take far too much time to me to dig below reencode_string().

Also, I am really worried for buffer overflows here. Looking at
the code, it seems like headers from input mails are passed to
decode_header(), which itself calls decode_header_bq() which in
turns calls convert_to_utf8() with input and output buffers of
256 bytes. Same for mailinfo() which calls check_header_line()
with <line> which has been filled up to sizeof(line) (1000 bytes).

Unless I'm mistaken, encoding from latin1 to utf-8 can grow the
string by a 5-to-1 ratio. So this means that the blind strcpy()
in convert_to_utf8() could overflow on user-crafted mail headers.
Maybe I overlooked something :-/

Regards,
Willy