git-diff in a worktree is an order of magnitude slower?

git-diff in a worktree is an order of magnitude slower?

[D. Ben Knoble]

Hello all,

I'd like to report and offer to help fix what I view as a serious performance
bug:

    "git diff --no-ext-diff --quiet" performs about ~10x slower in a secondary
    worktree than in the main worktree.

Fortunately, this doesn't seem to extend to "--cached" (and "--no-ext-diff" and
"--quiet" are probably both red-herrings, since it _does_ extend to plain "git
diff").

Here's a short demo in Git:

    # git switch -d v2.54.0
    # ninja -C build # where my meson-generated build dir is
    # git worktree add --detach ../perf-test v2.54.0
    # hyperfine -N --warmup 10 './build/bin-wrappers/git diff'
    Benchmark 1: ./build/bin-wrappers/git diff
      Time (mean ± σ):       3.4 ms ±   0.5 ms    [User: 4.2 ms, System: 3.9 ms]
      Range (min … max):     2.5 ms …   5.8 ms    677 runs
    # pushd ../perf-test
    # hyperfine -N --warmup 10 '../git/build/bin-wrappers/git diff'
    Benchmark 1: ../git/build/bin-wrappers/git diff
      Time (mean ± σ):     223.3 ms ±  10.5 ms    [User: 210.4 ms,
System: 19.1 ms]
      Range (min … max):   213.5 ms … 243.9 ms    13 runs

I've had a similar experience at $DAYJOB, where a large repo takes ~6ms for the
former and ~650ms for the latter. I noticed because the Bash prompt functions
execute "git diff --no-ext-diff --quiet", and that was (AFAICT) the largest
culprit for a slow shell prompt in a worktree. To squelch that from the prompt,
I have to go down the rabbit hole of the worktree config extension, so I figured
better to fix the slow diff if possible anyway.

2 questions:

1. Is this known, and if so is anybody working on it?
2. How can I help identify problem areas?

A little more
-------------

I've reproduced this as far back as v2.50.0, which is as far back as I could get
the meson build to work with little effort (so I can't rule out that this is an
old regression).

Using "perf record -F 99 -g -- <bin-wrappers/git> diff" in both trees and then
"perf report":

- it looks like the main worktree spends most of it's time in preload_thread,
  threaded_has_symlink_leading_path, lstat_cache…
- the worktree spends a lot more time in ie_match_stat, ce_modified_check_fs,
  ce_compare_data, index_fd, would_convert_to_git_filter_fd…

Here's the relevant "perf stat":

main tree:

 Performance counter stats for './build/bin-wrappers/git diff':

                 0      context-switches:u               #      0,0
cs/sec  cs_per_second
                 0      cpu-migrations:u                 #      0,0
migrations/sec  migrations_per_second
               967      page-faults:u                    #  65036,4
faults/sec  page_faults_per_second
             14,87 msec task-clock:u                     #      0,3
CPUs  CPUs_utilized
            48 616      branch-misses:u                  #      3,2 %
branch_miss_rate         (57,19%)
         3 571 630      branches:u                       #    240,2
M/sec  branch_frequency
        13 635 411      cpu-cycles:u                     #      0,9
GHz  cycles_frequency
        22 120 068      instructions:u                   #      1,9
instructions  insn_per_cycle  (85,61%)
         3 634 065      stalled-cycles-frontend:u        #     0,28
frontend_cycles_idle        (9,56%)

       0,006860098 seconds time elapsed

       0,001364000 seconds user
       0,015157000 seconds sys

worktree:

 Performance counter stats for '../git/build/bin-wrappers/git diff':

                 0      context-switches:u               #      0,0
cs/sec  cs_per_second
                 0      cpu-migrations:u                 #      0,0
migrations/sec  migrations_per_second
             1 585      page-faults:u                    #   5058,0
faults/sec  page_faults_per_second
            313,37 msec task-clock:u                     #      0,9
CPUs  CPUs_utilized
         2 481 188      branch-misses:u                  #      1,5 %
branch_miss_rate         (48,94%)
       168 664 155      branches:u                       #    538,2
M/sec  branch_frequency     (51,21%)
     1 004 095 217      cpu-cycles:u                     #      3,2
GHz  cycles_frequency       (67,74%)
     3 864 851 223      instructions:u                   #      3,9
instructions  insn_per_cycle  (52,73%)
        70 755 234      stalled-cycles-frontend:u        #     0,07
frontend_cycles_idle        (49,29%)

       0,306707634 seconds time elapsed

       0,269027000 seconds user
       0,045512000 seconds sys

My observations:
- the worktree has ~twice as many page faults and
- executes ~150 times as many instructions (3.8b compared to 23m).

(When I try to run some "perf" stats as root to access other counters, like
syscalls, "git diff" in the worktree says "not a git repository", so I'm not
counting the actual behavior. Ditto with DTrace.)

PS I almost CC'd Peff and Patrick, whose names stood out in "git
shortlog builtin/{worktree,diff}* object-file* | sort -t\( -k2 -g",
but decided they'd be their own best judge of whether they can
understand what's going on? :)

-- 
D. Ben Knoble

Re: git-diff in a worktree is an order of magnitude slower?

[Jeff King]

On Mon, Jun 08, 2026 at 07:36:45PM -0400, D. Ben Knoble wrote:

> I'd like to report and offer to help fix what I view as a serious performance
> bug:
> 
>     "git diff --no-ext-diff --quiet" performs about ~10x slower in a secondary
>     worktree than in the main worktree.

Hmm, I get the opposite effect: it is much faster in the worktree!

I did:

  git clone /path/to/linux.git
  git -C linux worktree add --detach ../wt
  hyperfine -L dir linux,wt 'git -C {dir} diff'

which yielded:

  Benchmark 1: git -C linux diff
    Time (mean ± σ):     188.9 ms ±   2.5 ms    [User: 166.4 ms, System: 130.7 ms]
    Range (min … max):   185.5 ms … 194.8 ms    16 runs
  
  Benchmark 2: git -C wt diff
    Time (mean ± σ):      20.0 ms ±   1.5 ms    [User: 23.4 ms, System: 103.5 ms]
    Range (min … max):    17.2 ms …  24.6 ms    132 runs
  
  Summary
    git -C wt diff ran
      9.43 ± 0.71 times faster than git -C linux diff

Running:

  perf record -g git -C wt --no-pager diff
  perf record -g git -C linux --no-pager diff
  perf diff

implies that the slow case is spending a lot more time computing sha1s.
Which implies that the entries are stat dirty. And indeed, if I run:

  git -C linux update-index --refresh

now they both take ~20ms.

I wonder if it's just a racy-git problem? Many files are written in the
same second as the index, so they end up with the same mtimes, and we
have to err on the side of checking the contents.

See Documentation/technical/racy-git.adoc for a larger discussion.

So it is not really about worktrees at all, but just "bad luck" in
generating that initial index (that goes away next time you actually
make an index update that rewrites the whole thing).

I'd have thought USE_NSEC was the default these days, but looks like it
isn't? Try building with that and I'll bet it goes away entirely.

> PS I almost CC'd Peff and Patrick, whose names stood out in "git
> shortlog builtin/{worktree,diff}* object-file* | sort -t\( -k2 -g",
> but decided they'd be their own best judge of whether they can
> understand what's going on? :)

You might be interested in "git shortlog -ns". :)

-Peff

Re: git-diff in a worktree is an order of magnitude slower?

[D. Ben Knoble]

On Mon, Jun 8, 2026 at 8:11 PM Jeff King <peff@peff.net> wrote:
>
> On Mon, Jun 08, 2026 at 07:36:45PM -0400, D. Ben Knoble wrote:
>
> > I'd like to report and offer to help fix what I view as a serious performance
> > bug:
> >
> >     "git diff --no-ext-diff --quiet" performs about ~10x slower in a secondary
> >     worktree than in the main worktree.
>
> Hmm, I get the opposite effect: it is much faster in the worktree!
>
> I did:
>
>   git clone /path/to/linux.git
>   git -C linux worktree add --detach ../wt
>   hyperfine -L dir linux,wt 'git -C {dir} diff'
>
> which yielded:
>
>   Benchmark 1: git -C linux diff
>     Time (mean ± σ):     188.9 ms ±   2.5 ms    [User: 166.4 ms, System: 130.7 ms]
>     Range (min … max):   185.5 ms … 194.8 ms    16 runs
>
>   Benchmark 2: git -C wt diff
>     Time (mean ± σ):      20.0 ms ±   1.5 ms    [User: 23.4 ms, System: 103.5 ms]
>     Range (min … max):    17.2 ms …  24.6 ms    132 runs
>
>   Summary
>     git -C wt diff ran
>       9.43 ± 0.71 times faster than git -C linux diff
>
> Running:
>
>   perf record -g git -C wt --no-pager diff
>   perf record -g git -C linux --no-pager diff
>   perf diff
>
> implies that the slow case is spending a lot more time computing sha1s.
> Which implies that the entries are stat dirty. And indeed, if I run:
>
>   git -C linux update-index --refresh
>
> now they both take ~20ms.

Ah, TIL about --refresh. I suppose it could be nice if "git diff"
updated the index in this way, but that sounds like a band-aid. Maybe
creating a fresh worktree should do the equivalent to make sure it's
considered "fresh"?

(This also dropped my timings down to normal.)

At $DAYJOB, I _think_ some version of "git restore <stuff>" ended up
also updating the index.

> I wonder if it's just a racy-git problem? Many files are written in the
> same second as the index, so they end up with the same mtimes, and we
> have to err on the side of checking the contents.
>
> See Documentation/technical/racy-git.adoc for a larger discussion.
>
> So it is not really about worktrees at all, but just "bad luck" in
> generating that initial index (that goes away next time you actually
> make an index update that rewrites the whole thing).

Ah, that makes sense! I'm familiar with the raciness but didn't expect it here.

> I'd have thought USE_NSEC was the default these days, but looks like it
> isn't? Try building with that and I'll bet it goes away entirely.

Thanks, I'll take a look.

I can see on my Macbook that at least Meson does automatically set
either USE_ST_TIMESPEC or NO_NSEC automatically, but has no option to
enabled USE_NSEC and try that. I can probably write that patch (which
I'll do to test), and I can send it along with the "worktree add
should refresh the index" if you think that's an appropriate thing to
do.

> > PS I almost CC'd Peff and Patrick, whose names stood out in "git
> > shortlog builtin/{worktree,diff}* object-file* | sort -t\( -k2 -g",
> > but decided they'd be their own best judge of whether they can
> > understand what's going on? :)
>
> You might be interested in "git shortlog -ns". :)
>
> -Peff

Phew! Yeah, that's much nicer. Thanks! (When typing this out for
email, I even left out the "grep '^[^[:space:]]' |" filter :P)

-- 
D. Ben Knoble

Re: git-diff in a worktree is an order of magnitude slower?

[Jeff King]

On Tue, Jun 09, 2026 at 01:15:11PM -0400, D. Ben Knoble wrote:

> > Which implies that the entries are stat dirty. And indeed, if I run:
> >
> >   git -C linux update-index --refresh
> >
> > now they both take ~20ms.
> 
> Ah, TIL about --refresh. I suppose it could be nice if "git diff"
> updated the index in this way, but that sounds like a band-aid. Maybe
> creating a fresh worktree should do the equivalent to make sure it's
> considered "fresh"?

I think "git diff" _does_ refresh the index internally (that's what
takes so long!). I thought we then wrote out the result, but maybe we
don't notice that it needs an update for some reason?

I'm pretty sure "git status" does something similar, though running it
in a slow working tree _does_ seem to make things faster. Maybe it's
more aggressive about doing the update.

I don't think that refreshing after making a worktree would help. The
problem is one of timestamps: we just wrote an index (so it _should_ be
totally up to date), but we err on the side of caution for some entries
because the file timestamps and the index timestamp are the same. So
what makes it "work" is that one second passed between writing those
files and running "update-index". If you ran it from the worktree
command automatically, it might all still happen in the same second.

And of course, it's not just worktrees. Any time we checkout we may
suffer from this problem, though initial clones and worktree creation
will write more files than most.


> At $DAYJOB, I _think_ some version of "git restore <stuff>" ended up
> also updating the index.

Yep, that would make sense. Any index write (after the second-hand
ticks) will make it go away, since it means updating the mtime of the
index.

> > I'd have thought USE_NSEC was the default these days, but looks like it
> > isn't? Try building with that and I'll bet it goes away entirely.
> 
> Thanks, I'll take a look.
> 
> I can see on my Macbook that at least Meson does automatically set
> either USE_ST_TIMESPEC or NO_NSEC automatically, but has no option to
> enabled USE_NSEC and try that. I can probably write that patch (which
> I'll do to test), and I can send it along with the "worktree add
> should refresh the index" if you think that's an appropriate thing to
> do.

I think NO_NSEC is about not looking at the nsec fields of stat structs
(since they might not exist). But we don't actually use them for stat
matching unless USE_NSEC is set.

I guess the distinction goes back to c06ff4908b (Record ns-timestamps if
possible, but do not use it without USE_NSEC, 2009-03-04), which details
some reasons you might not want USE_NSEC. Feels like it ought to be a
run-time config, though, and maybe even something that gets auto-probed
by git-init.

Definitely not an area I have looked at much, though, nor thought hard
about. So there might be gotchas. :)

-Peff

Re: git-diff in a worktree is an order of magnitude slower?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I guess the distinction goes back to c06ff4908b (Record ns-timestamps if
> possible, but do not use it without USE_NSEC, 2009-03-04), which details
> some reasons you might not want USE_NSEC. Feels like it ought to be a
> run-time config, though, and maybe even something that gets auto-probed
> by git-init.

I thought for a bit but didn't think of a clean way to auto-probe if
a filesystem loses nanosecond-precision part of .st_Xtime when
"metadata is flushed and later read back in" with reasonable
overhead.  I do not think we want to trigger system-wide sync and/or
dropping of buffer cache ;-)

> Definitely not an area I have looked at much, though, nor thought hard
> about. So there might be gotchas. :)
>
> -Peff

Re: git-diff in a worktree is an order of magnitude slower?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

On 2026-06-11 at 17:43:52, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
> 
> > I guess the distinction goes back to c06ff4908b (Record ns-timestamps if
> > possible, but do not use it without USE_NSEC, 2009-03-04), which details
> > some reasons you might not want USE_NSEC. Feels like it ought to be a
> > run-time config, though, and maybe even something that gets auto-probed
> > by git-init.
> 
> I thought for a bit but didn't think of a clean way to auto-probe if
> a filesystem loses nanosecond-precision part of .st_Xtime when
> "metadata is flushed and later read back in" with reasonable
> overhead.  I do not think we want to trigger system-wide sync and/or
> dropping of buffer cache ;-)

We could have `git update-index` take options like it does for
`--untracked-cache` and `--no-untracked-cache` to control these for
people who want them.  For instance, I know what operating system and
file system I'm using (Linux with btrfs), so if I know that option is
safe, I can enable it at runtime and reap the benefits.

We could even have `--test-use-nsec` to perform a `uname` and `statfs`
call to determine whether this is a known safe configuration if probing
is not possible.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]