APFS, mmap, and git diff reporting no changes when there are changes?

APFS, mmap, and git diff reporting no changes when there are changes?

[Harold Hunt]

Hi All - I used to maintain Cygwin/X (the X Server for Cygwin on
Windows) 20 years ago... it's been a while since I've used a mailing
list.

I'm working on my own implementation of copy on write cloning for git
on a couple Linux fs options and APFS on Mac.  While working on this I
created tests around mmap'd files and how to handle sharing them when
creating worktrees with clones.

I discovered that mmap'd files on Mac can cause existing git to
misreport no changes on various commands, while tools that read the
bytes of the files will see the changed bytes.  I could not find any
discussion of this being an intentional tradeoff of speed over
correctness for git on APFS.  I don't believe this is a security
problem.  However, any tool that mmaps a file, changes it, does not
call msync(), and holds that mmap open will cause this misreport to
continue until munmap() is called, at which point the ctime of the
file changes (mtime does not change on this path).

One could argue that not calling msync() (which updates both ctime and
mtime) is bad form and that git not reporting the changes in this case
is an acceptable tradeoff for performance reasons.  However, on Linux,
git correctly detects the changes via the mmap even without calling
msync().

One could also argue that this is Apple's bug or deficiency.  But that
could take months / years to resolve?

I'm writing to ask a few questions before either extracting and
submitting just the tests or carefully vetting the fixes and preparing
my own PRs:
1. Is this already known or is this nothing?
2. How far do we want to go for correctness?
3. Does anyone else have experience with this on APFS and have some advice?

I have an initial LLM-written PR for this on my fork (I am NOT asking
for review on this... if you want to look, look, but if you don't want
to: please don't): https://github.com/huntharo/git/pull/14

The impact outside of APFS file systems is zero.  The trade off on
APFS depends on how many files are mmap'd and how large they are. If
no files are ever mmap'd by any tools, ever, then the cost can be
zero.  The technique I'm using looks at generation number being 0 as
the indication that we have to examine the file contents as the
generation number drops to 0 while mapped.

There is a related follow-on problem: changing a file via mmap/munmap
(or any other mechanism that updates ctime but not mtime) in the same
second as the last commit will cause git to continue to report no
changes even with the file unmapped.

Thanks for any input you can provide here.  If these aren't really
considered to be a problem then I'll leave them unsolved in my CoW
clone patches when I'm ready to submit.

Harold

Re: APFS, mmap, and git diff reporting no changes when there are changes?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3521 bytes --]

On 2026-11-09 at 22:05:55, Harold Hunt wrote:
> Hi All - I used to maintain Cygwin/X (the X Server for Cygwin on
> Windows) 20 years ago... it's been a while since I've used a mailing
> list.
> 
> I'm working on my own implementation of copy on write cloning for git
> on a couple Linux fs options and APFS on Mac.  While working on this I
> created tests around mmap'd files and how to handle sharing them when
> creating worktrees with clones.
> 
> I discovered that mmap'd files on Mac can cause existing git to
> misreport no changes on various commands, while tools that read the
> bytes of the files will see the changed bytes.  I could not find any
> discussion of this being an intentional tradeoff of speed over
> correctness for git on APFS.  I don't believe this is a security
> problem.  However, any tool that mmaps a file, changes it, does not
> call msync(), and holds that mmap open will cause this misreport to
> continue until munmap() is called, at which point the ctime of the
> file changes (mtime does not change on this path).
> 
> One could argue that not calling msync() (which updates both ctime and
> mtime) is bad form and that git not reporting the changes in this case
> is an acceptable tradeoff for performance reasons.  However, on Linux,
> git correctly detects the changes via the mmap even without calling
> msync().

Git doesn't use mmap on files that change in place, only on
content-addressed files like packs, indexes, and (for interoperability
code) object maps, all of which have hashes in their names, or files
which are replaced by rename(2), like the index.

In general, it's not safe to use mmap on files that can change in place
because if you access beyond the end of the file, you can get SIGBUS,
but you don't know if the file may have been modified at any point, so
you effectively have to deal with crashing anywhere in your program.

> One could also argue that this is Apple's bug or deficiency.  But that
> could take months / years to resolve?

This behaviour is allowed by POSIX:

    The last data modification and last file status change timestamps of
    a file that is mapped with MAP_SHARED and PROT_WRITE shall be marked
    for update at some point in the interval between a write reference
    to the mapped region and the next call to msync() with MS_ASYNC or
    MS_SYNC for that portion of the file by any process.

However, Git relies on the operating system to reflect an accurate
modification time.  If you write files to disk with mmap, then you're
responsible for calling msync() in a timely manner.  However, as I said,
that's extremely improvident for things like files in the working tree,
which is why text editors don't use mmap.

> I'm writing to ask a few questions before either extracting and
> submitting just the tests or carefully vetting the fixes and preparing
> my own PRs:
> 1. Is this already known or is this nothing?
> 2. How far do we want to go for correctness?
> 3. Does anyone else have experience with this on APFS and have some advice?
> 
> I have an initial LLM-written PR for this on my fork (I am NOT asking
> for review on this... if you want to look, look, but if you don't want
> to: please don't): https://github.com/huntharo/git/pull/14

Please read the entirety of SubmittingPatches, including
https://git-scm.com/docs/SubmittingPatches#ai, before sending that in.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA
Speaking for myself only

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]

Re: APFS, mmap, and git diff reporting no changes when there are changes?

[Johannes Sixt]

Am 09.11.26 um 23:38 schrieb brian m. carlson:
> On 2026-11-09 at 22:05:55, Harold Hunt wrote:
>> One could also argue that this is Apple's bug or deficiency.  But that
>> could take months / years to resolve?
> 
> This behaviour is allowed by POSIX:
> 
>     The last data modification and last file status change timestamps of
>     a file that is mapped with MAP_SHARED and PROT_WRITE shall be marked
>     for update at some point in the interval between a write reference
>     to the mapped region and the next call to msync() with MS_ASYNC or
>     MS_SYNC for that portion of the file by any process.
> 
> However, Git relies on the operating system to reflect an accurate
> modification time.  If you write files to disk with mmap, then you're
> responsible for calling msync() in a timely manner.  However, as I said,
> that's extremely improvident for things like files in the working tree,
> which is why text editors don't use mmap.

Not to mention that our mmap emulation for Windows cannot reproduce the
Linux behavior (timely update of timestamps) and is more like the macOS
behavior (delayed update of timestamps). We cannot have an accurate
(atomic) msync emulation, either, because not even FlushViewOfFile[1]
updates the timestamp.

[1]
https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-flushviewoffile

-- Hannes