Re: [PATCH v3 1/1] Support working-tree-encoding "UTF-16LE-BOM"

["Torsten Bögershausen" <tboegi@web.de>]

On Wed, Jan 30, 2019 at 10:24:44AM -0500, Jason Pyeron wrote:
> > -----Original Message-----
> > From: git-owner@vger.kernel.org <git-owner@vger.kernel.org> On Behalf Of
> > tboegi@web.de
> > Sent: Wednesday, January 30, 2019 10:02 AM
> > To: git@vger.kernel.org; adrigibal@gmail.com
> > Cc: Torsten Bögershausen <tboegi@web.de>
> > Subject: [PATCH v3 1/1] Support working-tree-encoding "UTF-16LE-BOM"
> >
> > From: Torsten Bögershausen <tboegi@web.de>
> >
> > Users who want UTF-16 files in the working tree set the .gitattributes
> > like this:
> > test.txt working-tree-encoding=UTF-16
> >
> > The unicode standard itself defines 3 allowed ways how to encode UTF-16.
> > The following 3 versions convert all back to 'g' 'i' 't' in UTF-8:
> >
> > a) UTF-16, without BOM, big endian:
> > $ printf "\000g\000i\000t" | iconv -f UTF-16 -t UTF-8 | od -c
> > 0000000    g   i   t
> >
> > b) UTF-16, with BOM, little endian:
> > $ printf "\377\376g\000i\000t\000" | iconv -f UTF-16 -t UTF-8 | od -c
> > 0000000    g   i   t
> >
> > c) UTF-16, with BOM, big endian:
> > $ printf "\376\377\000g\000i\000t" | iconv -f UTF-16 -t UTF-8 | od -c
> > 0000000    g   i   t
> >
> > Git uses libiconv to convert from UTF-8 in the index into ITF-16 in the
> > working tree.
> > After a checkout, the resulting file has a BOM and is encoded in "UTF-16",
> > in the version (c) above.
> > This is what iconv generates, more details follow below.
> >
> > iconv (and libiconv) can generate UTF-16, UTF-16LE or UTF-16BE:
> >
> > d) UTF-16
> > $ printf 'git' | iconv -f UTF-8 -t UTF-16 | od -c
> > 0000000  376 377  \0   g  \0   i  \0   t
> >
> > e) UTF-16LE
> > $ printf 'git' | iconv -f UTF-8 -t UTF-16LE | od -c
> > 0000000    g  \0   i  \0   t  \0
> >
> > f)  UTF-16BE
> > $ printf 'git' | iconv -f UTF-8 -t UTF-16BE | od -c
> > 0000000   \0   g  \0   i  \0   t
> >
> > There is no way to generate version (b) from above in a Git working tree,
> > but that is what some applications need.
> > (All fully unicode aware applications should be able to read all 3
> > variants,
> > but in practise we are not there yet).
> >
> > When producing UTF-16 as an output, iconv generates the big endian version
> > with a BOM. (big endian is probably chosen for historical reasons).
> >
> > iconv can produce UTF-16 files with little endianess by using "UTF-16LE"
> > as encoding, and that file does not have a BOM.
> >
> > Not all users (especially under Windows) are happy with this.
> > Some tools are not fully unicode aware and can only handle version (b).
> >
> > Today there is no way to produce version (b) with iconv (or libiconv).
> > Looking into the history of iconv, it seems as if version (c) will
> > be used in all future iconv versions (for compatibility reasons).
>
>
> Reading the RFC 2781 section 3.3:
>
>    Text in the "UTF-16BE" charset MUST be serialized with the octets
>    which make up a single 16-bit UTF-16 value in big-endian order.
>    Systems labelling UTF-16BE text MUST NOT prepend a BOM to the text.
>
>    Text in the "UTF-16LE" charset MUST be serialized with the octets
>    which make up a single 16-bit UTF-16 value in little-endian order.
>    Systems labelling UTF-16LE text MUST NOT prepend a BOM to the text.
>
> I opened a bug with libiconv... https://savannah.gnu.org/bugs/index.php?55609
>

UTF-16 may be a), b) or c) from above.
Every unicode compliant system should be able to read all 3 of them.

When writing, the system/application/converter is free to choose one of those.
Probably out of historical reason, big endian is preferred (in iconv),
and to be helpful to systems/applications a BOM is written in the beginning.
This is according to the RFC, why do you think that this is a bug ?