Potential bug: Git instantly converts utf-8-bom to utf-8

Potential bug: Git instantly converts utf-8-bom to utf-8

[messester]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

How this occurs:
When I clone the git repository https://github.com/NinekoTheCat/infinilore.cs.git into an empty directory and run git status it shows changes where all the files are deleted and then added again.
The difference seems to be in the first line as opening it in a text editor aware of encodings shows the change from utf-8-bom to utf-8.
(Attached to this email is an example patch file).

My git configuration in the directory is:
---
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=REDACT.crt
core.autocrlf=true
core.fscache=true
core.symlinks=false
pull.rebase=false
credential.helper=manager
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=main
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
user.name=REDACT
user.email=REDACT
user.signingkey=4D973217092A925B
core.editor="C:\Users\cnots\AppData\Local\Programs\Microsoft VS Code\bin\code" --wait
safe.directory=REDACT
commit.gpgsign=true
gpg.program=C:\Program Files (x86)\GnuPG\bin\gpg.exe
credential.https://github.com.helper=
credential.https://github.com.helper=!REDACT
credential.https://gist.github.com.helper=
credential.https://gist.github.com.helper=!REDACT
---
and the version of git I use is 2.49.0.windows.1.

I'd be very very grateful if anyone could explain what's going on and have a solution.

[-- Attachment #2: patch.cat --]
[-- Type: application/vnd.ms-pki.seccat, Size: 247906 bytes --]

Re: Potential bug: Git instantly converts utf-8-bom to utf-8

[Lucas Seiki Oshiro]

> When I clone the git repository https://github.com/NinekoTheCat/infinilore.cs.git into an empty directory and run git status it shows changes where all the files are deleted and then added again.

I tried here. I'm in a Mac, it shows that several file changes.

> The difference seems to be in the first line as opening it in a text editor aware of encodings shows the change from utf-8-bom to utf-8.

Given the warning messages it looks like it is more related
to CRLF/LF than to UTF-8.

At least here, for example, the listed modified '*.cs' files
are the ones that ends their lines in CRLF. This is, running:

```
file $(git ls-files '*.cs') | grep CRLF | cut -d ':' -f1 | sort
```

shows me the same files as

```
git status '*.cs'
```

You can also see that are some '*.cs' files that are not
CRLF-terminated by running:

```
file $(git ls-files '*.cs') | grep -v CRLF | sort
```

> I'd be very very grateful if anyone could explain what's going on and have a solution.<patch.cat>

Perhaps a solution is:

- Define a CRLF or LF as a standard
- Convert all the files that don't follow the standard that you
  have chosen to the one that you have chosen
- Declare it in the .gitattributes file

Hope that it helps you!

Re: Potential bug: Git instantly converts utf-8-bom to utf-8

[Torsten Bögershausen]

On Mon, Apr 14, 2025 at 02:19:05PM -0300, Lucas Seiki Oshiro wrote:
>
> > When I clone the git repository https://github.com/NinekoTheCat/infinilore.cs.git into an empty directory and run git status it shows changes where all the files are deleted and then added again.
>
> I tried here. I'm in a Mac, it shows that several file changes.
>
> > The difference seems to be in the first line as opening it in a text editor aware of encodings shows the change from utf-8-bom to utf-8.
>
> Given the warning messages it looks like it is more related
> to CRLF/LF than to UTF-8.
>
> At least here, for example, the listed modified '*.cs' files
> are the ones that ends their lines in CRLF. This is, running:
>
> ```
> file $(git ls-files '*.cs') | grep CRLF | cut -d ':' -f1 | sort
> ```
>
> shows me the same files as
>
> ```
> git status '*.cs'
> ```
>
> You can also see that are some '*.cs' files that are not
> CRLF-terminated by running:
>
> ```
> file $(git ls-files '*.cs') | grep -v CRLF | sort
> ```
>
> > I'd be very very grateful if anyone could explain what's going on and have a solution.<patch.cat>
>
> Perhaps a solution is:
>
> - Define a CRLF or LF as a standard
> - Convert all the files that don't follow the standard that you
>   have chosen to the one that you have chosen
> - Declare it in the .gitattributes file
>
> Hope that it helps you!

I think that this is already done :-)
However, the repo must be normalized:

git add --renormalize .

will do that.

Re: Potential bug: Git instantly converts utf-8-bom to utf-8

[Junio C Hamano]

Torsten Bögershausen <tboegi@web.de> writes:

> I think that this is already done :-)
> However, the repo must be normalized:
>
> git add --renormalize .
>
> will do that.

Thanks ;-).