BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Sjur Moshagen]

Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)
git clone https://github.com/giellalt/lang-sma

What did you expect to happen? (Expected behavior)
Clone to be clean, as reported by git status

What happened instead? (Actual behavior)
git status reported four changed files

What's different between what you expected and what actually happened?
Nothing except those four files

Anything else you want to add:
This only happens on an M2 Macbook Pro. With Apple's git (1.37.1), a huge number of files were reported as modified.

Please review the rest of the bug report below.
You can delete any lines you don't wish to share.


[System Info]
git version:
git version 2.40.0
cpu: arm64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon
uname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:46 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6020 arm64
compiler info: clang: 14.0.0 (clang-1400.0.29.202)
libc info: no libc information available
$SHELL (typically, interactive shell): /bin/zsh


[Enabled Hooks]

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Emily Shaffer]

On Wed, Mar 22, 2023 at 9:38 AM Sjur Moshagen <sjurnm@mac.com> wrote:
>
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
>
> What did you do before the bug happened? (Steps to reproduce your issue)
> git clone https://github.com/giellalt/lang-sma
>
> What did you expect to happen? (Expected behavior)
> Clone to be clean, as reported by git status
>
> What happened instead? (Actual behavior)
> git status reported four changed files

It would help to know which files, or what the difference was, so we
have a chance to figure out what happens without having to go and buy
a Macbook :)

>
> What's different between what you expected and what actually happened?
> Nothing except those four files
>
> Anything else you want to add:
> This only happens on an M2 Macbook Pro. With Apple's git (1.37.1), a huge number of files were reported as modified.
>
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
>
>
> [System Info]
> git version:
> git version 2.40.0
> cpu: arm64
> no commit associated with this build
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> feature: fsmonitor--daemon
> uname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:46 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6020 arm64
> compiler info: clang: 14.0.0 (clang-1400.0.29.202)
> libc info: no libc information available
> $SHELL (typically, interactive shell): /bin/zsh
>
>
> [Enabled Hooks]
>

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Sjur Moshagen]

22.3.2023 kl. 17:46 skrev Emily Shaffer <nasamuffin@google.com>:
> 
> On Wed, Mar 22, 2023 at 9:38 AM Sjur Moshagen <sjurnm@mac.com> wrote:
>> 
>> Thank you for filling out a Git bug report!
>> Please answer the following questions to help us understand your issue.
>> 
>> What did you do before the bug happened? (Steps to reproduce your issue)
>> git clone https://github.com/giellalt/lang-sma
>> 
>> What did you expect to happen? (Expected behavior)
>> Clone to be clean, as reported by git status
>> 
>> What happened instead? (Actual behavior)
>> git status reported four changed files
> 
> It would help to know which files, or what the difference was, so we
> have a chance to figure out what happens without having to go and buy
> a Macbook :)

Yes, of course :)

It only happens with binary files (.docx and .pdf files), which got me suspicious. It turned out that updating the .gitattributes file fixed the issue, cf

https://github.com/giellalt/lang-sma/commit/fb5b56fa9fa7f25be1a1d7190bbb4d12b5a908fd#diff-618cd5b83d62060ba3d027e314a21ceaf75d36067ff820db126642944145393e

What is still a mystery to me is that there has been no issues with cloning files on Intel Macs (git version 2.39.2), or on Linux (git version 2.34.1), even with the old .gitattributes file - ie the cloned repos were clean on these systems. Only on M2 (possibly also M1) Macs did this issue pop up.

Please tell me if there is more I can do to help.

Regards,
Sjur

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Kristoffer Haugsbakk]

On Wed, Mar 22, 2023, at 18:29, Sjur Moshagen wrote:
> [snip]
>
> Yes, of course :)
>
> It only happens with binary files (.docx and .pdf files), which got me 
> suspicious. It turned out that updating the .gitattributes file fixed 
> the issue, cf
>
> https://github.com/giellalt/lang-sma/commit/fb5b56fa9fa7f25be1a1d7190bbb4d12b5a908fd#diff-618cd5b83d62060ba3d027e314a21ceaf75d36067ff820db126642944145393e
>
> What is still a mystery to me is that there has been no issues with 
> cloning files on Intel Macs (git version 2.39.2), or on Linux (git 
> version 2.34.1), even with the old .gitattributes file - ie the cloned 
> repos were clean on these systems. Only on M2 (possibly also M1) Macs 
> did this issue pop up.
>
> Please tell me if there is more I can do to help.
>
> Regards,
> Sjur

I still get a dirty working repository after a fresh clone (on Linux,
git version 2.40.0) on your commit 6425cb09a8d0 (Apply changes 127,
2023-03-22):

    $ git status
    On branch main
    Your branch is up to date with 'origin/main'.

    Changes not staged for commit:
      (use "git add <file>..." to update what will be committed)
      (use "git restore <file>..." to discard changes in working directory)
            modified:   "corp/SN\303\205SNINGEN 2014/43 uke/N\303\246ringsliv.doc"
            modified:   "corp/SN\303\205SNINGEN 2015/26 uke/\303\230lsamling.doc"
            modified:   "corp/SN\303\205SNINGEN 2015/38 uke /SAK1 -s\303\270rsamisk.doc"
            modified:   "corp/SN\303\205SNINGEN 2015/38 uke /SAK1.doc"
            modified:   "corp/Sn\303\245sningen 2012/20 uke/Artig med animasjon - samisk.doc"
            modified:   "corp/Sn\303\245sningen 2012/20 uke/Artig med animasjon.doc"
            modified:   "corp/Sn\303\245sningen 2012/4. uke/Mange vil l\303\246re s\303\270rs/Mange vil l\303\246re s\303\270rsamisk - saemien.doc"
            modified:   "corp/Sn\303\245sningen 2012/4. uke/Mange vil l\303\246re s\303\270rs/Mange vil l\303\246re s\303\270rsamisk.doc"
            modified:   "corp/Sn\303\245sningen 2012/4. uke/P\303\245 sameskolen/Det er p\303\245 sameskolen jeg f\303\270ler meg hjemme - samisk.doc"
            modified:   "corp/Sn\303\245sningen 2012/4. uke/P\303\245 sameskolen/Det er p\303\245 sameskolen jeg f\303\270ler meg hjemme.doc"
            modified:   "corp/Sn\303\245sningen 2013/50 uke/Dramatikk og kj\303\246rlighet - S\303\270rsamisk.doc"

    no changes added to commit (use "git add" and/or "git commit -a")

Note that this `doc` extension was not covered by your update.

But it becomes clean if I remove this line:

    $ git diff
    diff --git a/.gitattributes b/.gitattributes
    index bbd8c0db810d..3959fefadfc1 100644
    --- a/.gitattributes
    +++ b/.gitattributes
    @@ -2,7 +2,6 @@
     # DO NOT EDIT - the file is updated via the template system.

     # Some defaults:
    -* text   eol=lf
     *.png    binary
     *.jpg    binary
     *.jpeg   binary

-- 
Kristoffer Haugsbakk

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Sjur Moshagen]

22.3.2023 kl. 19:22 skrev Kristoffer Haugsbakk <kristofferhaugsbakk@fastmail.com>:
> 
> I still get a dirty working repository after a fresh clone (on Linux,
> git version 2.40.0) on your commit 6425cb09a8d0 (Apply changes 127,
> 2023-03-22):
> 
>    $ git status
>    On branch main
>    Your branch is up to date with 'origin/main'.
> 
>    Changes not staged for commit:
>      (use "git add <file>..." to update what will be committed)
>      (use "git restore <file>..." to discard changes in working directory)
>            modified:   "corp/SN\303\205SNINGEN 2014/43 uke/N\303\246ringsliv.doc"
>            modified:   "corp/SN\303\205SNINGEN 2015/26 uke/\303\230lsamling.doc"
>            modified:   "corp/SN\303\205SNINGEN 2015/38 uke /SAK1 -s\303\270rsamisk.doc"
>            modified:   "corp/SN\303\205SNINGEN 2015/38 uke /SAK1.doc"
>            modified:   "corp/Sn\303\245sningen 2012/20 uke/Artig med animasjon - samisk.doc"
>            modified:   "corp/Sn\303\245sningen 2012/20 uke/Artig med animasjon.doc"
>            modified:   "corp/Sn\303\245sningen 2012/4. uke/Mange vil l\303\246re s\303\270rs/Mange vil l\303\246re s\303\270rsamisk - saemien.doc"
>            modified:   "corp/Sn\303\245sningen 2012/4. uke/Mange vil l\303\246re s\303\270rs/Mange vil l\303\246re s\303\270rsamisk.doc"
>            modified:   "corp/Sn\303\245sningen 2012/4. uke/P\303\245 sameskolen/Det er p\303\245 sameskolen jeg f\303\270ler meg hjemme - samisk.doc"
>            modified:   "corp/Sn\303\245sningen 2012/4. uke/P\303\245 sameskolen/Det er p\303\245 sameskolen jeg f\303\270ler meg hjemme.doc"
>            modified:   "corp/Sn\303\245sningen 2013/50 uke/Dramatikk og kj\303\246rlighet - S\303\270rsamisk.doc"
> 
>    no changes added to commit (use "git add" and/or "git commit -a")
> 
> Note that this `doc` extension was not covered by your update.
> 
> But it becomes clean if I remove this line:
> 
>    $ git diff
>    diff --git a/.gitattributes b/.gitattributes
>    index bbd8c0db810d..3959fefadfc1 100644
>    --- a/.gitattributes
>    +++ b/.gitattributes
>    @@ -2,7 +2,6 @@
>     # DO NOT EDIT - the file is updated via the template system.
> 
>     # Some defaults:
>    -* text   eol=lf

Thanks for pointing this out - git status did not return these files for me. Also interesting that you got it on a Linux box, that has not happened to us, cf earlier email in this thread.

I will add this and other missing extensions.

Best,
Sjur

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Junio C Hamano]

"Kristoffer Haugsbakk" <kristofferhaugsbakk@fastmail.com> writes:

>             modified:   "corp/Sn\303\245sningen 2013/50 uke/Dramatikk og kj\303\246rlighet - S\303\270rsamisk.doc"
>
>     no changes added to commit (use "git add" and/or "git commit -a")
>
> Note that this `doc` extension was not covered by your update.
>
> But it becomes clean if I remove this line:
>
>     $ git diff
>     diff --git a/.gitattributes b/.gitattributes
>     index bbd8c0db810d..3959fefadfc1 100644
>     --- a/.gitattributes
>     +++ b/.gitattributes
>     @@ -2,7 +2,6 @@
>      # DO NOT EDIT - the file is updated via the template system.
>
>      # Some defaults:
>     -* text   eol=lf
>      *.png    binary
>      *.jpg    binary
>      *.jpeg   binary

The attribute file says "assume everything is text unless otherwise
noted below", which you probably should not lose.  Instead, I would
declare *.doc are also binary by appending a new entry at the end of
the file, if I were managing that project.

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Kristoffer Haugsbakk]

On Wed, Mar 22, 2023, at 20:18, Junio C Hamano wrote:
> The attribute file says "assume everything is text unless otherwise
> noted below", which you probably should not lose.

That was to illustrate the issue, not to propose a change. Sorry about
not being clear.

(But it also so happens to be the case that I know very little about 
`.gitattributes` :))

On Wed, Mar 22, 2023, at 19:47, Sjur Moshagen wrote:
> I will add this and other missing extensions.

By the way, when I first cloned the repository on commit 357c7cf01950
(text, 2023-03-21), those two binary `txt` files showed up as dirty for
me (before you removed them in 6b0acd78f9c9 (Remove accidentally added
files with uncomprehensible content, 2023-03-22)). I guess it was the
same for you.

-- 
Kristoffer Haugsbakk

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Sjur Moshagen]

22.3.2023 kl. 21:00 skrev Kristoffer Haugsbakk <kristofferhaugsbakk@fastmail.com>:
> 
> On Wed, Mar 22, 2023, at 20:18, Junio C Hamano wrote:
>> The attribute file says "assume everything is text unless otherwise
>> noted below", which you probably should not lose.

I have now added all missing binary file types I could find, and kept the default as text.

> On Wed, Mar 22, 2023, at 19:47, Sjur Moshagen wrote:
>> I will add this and other missing extensions.

Done in https://github.com/giellalt/lang-sma/commit/0f461c63ade00a623da2b2533fbe5ca4b986c1f0.

> By the way, when I first cloned the repository on commit 357c7cf01950
> (text, 2023-03-21), those two binary `txt` files showed up as dirty for
> me (before you removed them in 6b0acd78f9c9 (Remove accidentally added
> files with uncomprehensible content, 2023-03-22)). I guess it was the
> same for you.

Yes, it was, should be ok now.

Thanks to all for the help and friendly comments and suggestions.

Sjur Moshagen

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Jeff King]

On Wed, Mar 22, 2023 at 09:46:11AM -0700, Emily Shaffer wrote:

> On Wed, Mar 22, 2023 at 9:38 AM Sjur Moshagen <sjurnm@mac.com> wrote:
> >
> > Thank you for filling out a Git bug report!
> > Please answer the following questions to help us understand your issue.
> >
> > What did you do before the bug happened? (Steps to reproduce your issue)
> > git clone https://github.com/giellalt/lang-sma
> >
> > What did you expect to happen? (Expected behavior)
> > Clone to be clean, as reported by git status
> >
> > What happened instead? (Actual behavior)
> > git status reported four changed files
> 
> It would help to know which files, or what the difference was, so we
> have a chance to figure out what happens without having to go and buy
> a Macbook :)

Just a guess, but it probably has to do with Unicode normalization.

  $ git clone https://github.com/giellalt/lang-sma
  $ cd lang-sma
  $ git ls-files -z |
    perl -MUnicode::Normalize -0ne '
      chomp;
      print "$_\n" if NFD($_) ne $_;
    '
  corp/SNÅSNINGEN 2014/10 uke/Arbeidsliv - sørsamisk.docx
  corp/SNÅSNINGEN 2014/10 uke/Arbeidsliv.docx
  [...and so on, there are 540 entries...]

HFS silently normalized everything to NFD. On newer installs using APFS,
I think the rules are different, but I haven't followed the details.

I'm not sure if this is a red herring (after all, I'd expect 540 such
files on HFS) or if some of the files are somehow special for APFS (I
looked for obvious things like collisions after normalization, but
didn't see any). I agree it would help to know which 4 files. :)

-Peff

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Jeff King]

On Wed, Mar 22, 2023 at 01:33:51PM -0400, Jeff King wrote:

> On Wed, Mar 22, 2023 at 09:46:11AM -0700, Emily Shaffer wrote:
> 
> > On Wed, Mar 22, 2023 at 9:38 AM Sjur Moshagen <sjurnm@mac.com> wrote:
> > >
> > > Thank you for filling out a Git bug report!
> > > Please answer the following questions to help us understand your issue.
> > >
> > > What did you do before the bug happened? (Steps to reproduce your issue)
> > > git clone https://github.com/giellalt/lang-sma
> > >
> > > What did you expect to happen? (Expected behavior)
> > > Clone to be clean, as reported by git status
> > >
> > > What happened instead? (Actual behavior)
> > > git status reported four changed files
> > 
> > It would help to know which files, or what the difference was, so we
> > have a chance to figure out what happens without having to go and buy
> > a Macbook :)
> 
> Just a guess, but it probably has to do with Unicode normalization.

Heh. After reading the other response in this thread, I could not have
been further from the truth. :) Sorry for the noise.

-Peff

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Torsten Bögershausen]

On Wed, Mar 22, 2023 at 05:28:49PM +0100, Sjur Moshagen wrote:
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
>
> What did you do before the bug happened? (Steps to reproduce your issue)
> git clone https://github.com/giellalt/lang-sma
>
> What did you expect to happen? (Expected behavior)
> Clone to be clean, as reported by git status
>
> What happened instead? (Actual behavior)
> git status reported four changed files
>
> What's different between what you expected and what actually happened?
> Nothing except those four files
>
> Anything else you want to add:
> This only happens on an M2 Macbook Pro. With Apple's git (1.37.1), a huge number of files were reported as modified.
>
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
>
>
> [System Info]
> git version:
> git version 2.40.0
> cpu: arm64
> no commit associated with this build
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> feature: fsmonitor--daemon
> uname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:46 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6020 arm64
> compiler info: clang: 14.0.0 (clang-1400.0.29.202)
> libc info: no libc information available
> $SHELL (typically, interactive shell): /bin/zsh
>
>
> [Enabled Hooks]
>

As a general note, the current .gitattributes file can still be improved.
The general rule and recommendation would be to start the text attributes
definition by using a "catch all" rule, followed by files that need
special treatment.
The very first text/binary line would be
* text=auto
which will tell Git to auto-detect all files (and file types),
see below,
This basically prevents corruption of binary files.
which are not mentioned later.
Like README.md, LICENCE and so on.
Then there is a list of file extension, that are known to be binary,
you can add them as shown below.

===============================
# List of files to be included in GitHub statistics.
# DO NOT EDIT - the file is updated via the template system.

# Some defaults:
* text=auto
*.aif     binary
*.aiff    binary
*.docx    binary
*.fomabin binary
*.gz      binary
[snip]
*.sh text eol=lf

===================================
Now, lets look at the at the master branch of the repo:
commit 4e0d949dbbd13a4dd9285ae3dee63abc822b805a (HEAD -> main, origin/main, origin/HEAD)
Author: Sjur N Moshagen <sjurnm@mac.com>

Please run
git ls-files --eol
And you will see a line like this:
i/-text w/-text attr/text eol=lf	corp/SNÅSNINGEN 2014/43 uke/Næringsliv.doc

The "i/-text" will tell us thet Git identified the file, as commited into the repo,
as binary ("non-text").
The "w/-text" will tell us thet Git identified the file, as seen on disk,
in you working tree,  as binary ("non-text").
So everything looks fine, so far.
But the comes "attr/text eol=lf", wich tells Git to ignore its feeling,
and possible corrupt the file at the next commit, converting CRLF into LF.

As an exmple, the "corp/SNÅSNINGEN 2014/43 uke/Næringsliv.doc" will be corrupted.
It may or not be reported as changed. after a fresh clone. And others as well.
But in any case, it will be corrupted when you commit it.
Oops.

And why is that?
Because people had forgotten a line like
*.doc binary
for the doc files.

To put it in harsh words: I would call the .gitattributes file broken.
Please consider to change the default to auto, as pointed out by
others as well.

My version of a .gitattributes would look like this:
=================
# Some defaults:
# Stay on the safe type for files that may be added later
* text=auto
# shell files need lf
*.sh text eol=lf
# Add other file types, like *.gz

# Include in statistics, no syntax colour ATM, classified as text instead:
[snip]
====================

Re: BUGREPORT: Modified files on a fresh clone on M2 MacBook Pro

[Junio C Hamano]

Torsten Bögershausen <tboegi@web.de> writes:

> The general rule and recommendation would be to start the text attributes
> definition by using a "catch all" rule, followed by files that need
> special treatment.
> The very first text/binary line would be
> * text=auto
> which will tell Git to auto-detect all files (and file types),
> see below,
> This basically prevents corruption of binary files.
> which are not mentioned later.
> Like README.md, LICENCE and so on.
> Then there is a list of file extension, that are known to be binary,
> you can add them as shown below.

Thanks for writing this down clearly.

If I would add one thing to the general advice, it is that it
applies o those who are using Git primarily as their source code,
not binary asset, control system, which roughly translates to "for
most of the paths in the project, 'git diff' without using any
external diff driver is useful".  In such an environment,
"everything by default is text, and only selected few are not" would
work very well.  If on the other hand there are myriad of binary
asset of different types, it can become cumbersome to mark them "do
not try normalizing the eol, which is done to help cross platform
development" individually.