Re: APFS, mmap, and git diff reporting no changes when there are changes?

["brian m. carlson" <sandals@crustytoothpaste.net>]

[-- Attachment #1: Type: text/plain, Size: 3521 bytes --]

On 2026-11-09 at 22:05:55, Harold Hunt wrote:
> Hi All - I used to maintain Cygwin/X (the X Server for Cygwin on
> Windows) 20 years ago... it's been a while since I've used a mailing
> list.
> 
> I'm working on my own implementation of copy on write cloning for git
> on a couple Linux fs options and APFS on Mac.  While working on this I
> created tests around mmap'd files and how to handle sharing them when
> creating worktrees with clones.
> 
> I discovered that mmap'd files on Mac can cause existing git to
> misreport no changes on various commands, while tools that read the
> bytes of the files will see the changed bytes.  I could not find any
> discussion of this being an intentional tradeoff of speed over
> correctness for git on APFS.  I don't believe this is a security
> problem.  However, any tool that mmaps a file, changes it, does not
> call msync(), and holds that mmap open will cause this misreport to
> continue until munmap() is called, at which point the ctime of the
> file changes (mtime does not change on this path).
> 
> One could argue that not calling msync() (which updates both ctime and
> mtime) is bad form and that git not reporting the changes in this case
> is an acceptable tradeoff for performance reasons.  However, on Linux,
> git correctly detects the changes via the mmap even without calling
> msync().

Git doesn't use mmap on files that change in place, only on
content-addressed files like packs, indexes, and (for interoperability
code) object maps, all of which have hashes in their names, or files
which are replaced by rename(2), like the index.

In general, it's not safe to use mmap on files that can change in place
because if you access beyond the end of the file, you can get SIGBUS,
but you don't know if the file may have been modified at any point, so
you effectively have to deal with crashing anywhere in your program.

> One could also argue that this is Apple's bug or deficiency.  But that
> could take months / years to resolve?

This behaviour is allowed by POSIX:

    The last data modification and last file status change timestamps of
    a file that is mapped with MAP_SHARED and PROT_WRITE shall be marked
    for update at some point in the interval between a write reference
    to the mapped region and the next call to msync() with MS_ASYNC or
    MS_SYNC for that portion of the file by any process.

However, Git relies on the operating system to reflect an accurate
modification time.  If you write files to disk with mmap, then you're
responsible for calling msync() in a timely manner.  However, as I said,
that's extremely improvident for things like files in the working tree,
which is why text editors don't use mmap.

> I'm writing to ask a few questions before either extracting and
> submitting just the tests or carefully vetting the fixes and preparing
> my own PRs:
> 1. Is this already known or is this nothing?
> 2. How far do we want to go for correctness?
> 3. Does anyone else have experience with this on APFS and have some advice?
> 
> I have an initial LLM-written PR for this on my fork (I am NOT asking
> for review on this... if you want to look, look, but if you don't want
> to: please don't): https://github.com/huntharo/git/pull/14

Please read the entirety of SubmittingPatches, including
https://git-scm.com/docs/SubmittingPatches#ai, before sending that in.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA
Speaking for myself only

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]