[PATCH 0/2] status: stop holding index.lock while collecting

[Mihai Cosma <mcosma@gmail.com>]

Background "git status" runs from editors and shell prompts hold
index.lock for their whole run, so a "git add" or "git commit" run at
the same time fails. Patch 2 makes status take the lock only to write,
and only when it has something to write.

Patch 1 fixes a bug that patch 2 relies on but that exists on its own:
under index.skipHash (on with feature.manyFiles and in Scalar repos),
the check that stops status from overwriting a concurrently updated
index passes for any index, so status can silently revert a "git add",
also on 2.56.0. It applies to maint and can go in alone. There's a
reproduction below its "---".

This overlaps with Jun Wu's "Wait for the index lock instead of
failing" series, and the two fit together: his makes writers wait for
the lock, this makes status hold it rarely and briefly. They only
conflict on the one lock call in cmd_status(), which his series renames
to repo_try_hold_locked_index(). I'm happy to rebase onto it. On top of
his v2, the slowest "git add" in the clean case below went from 1020 ms
(one failure at his 1s timeout) to 160 ms.

Numbers on master, from a gcc checkout (124,311 files) on Linux, timing
200 "git add" runs while "git status" runs in a loop. "Active" touches
a tracked file before each status, so every status run writes the
index:

                       master      this series
  clean   failed       104/200     0/200
          p95 (ms)     114         128
  active  failed       123/200     47/200
          p95 (ms)     117         111

Failed runs on master return fast, which is why its timings look good.
In the active case status still holds the lock while it writes, about
76 ms for gcc's 14 MB index, and that's the part a short wait like
Jun's covers.

The script, run from the top of the checkout as "bash measure.sh clean"
or "bash measure.sh active":

  touched=$(git ls-files | head -1); echo x >measure-file
  ( end=$((SECONDS+60)); while [ $SECONDS -lt $end ]; do
      [ "$1" = active ] && touch "$touched"
      git status --porcelain >/dev/null 2>&1
    done ) &
  sleep 1
  for i in $(seq 200); do
      s=$(date +%s%N); git add measure-file 2>/dev/null; rc=$?
      echo $(( ($(date +%s%N) - s) / 1000000 )) $rc; sleep 0.05
  done | sort -n | awk '{t[NR]=$1; f+=($2!=0)} END {
      print "median", t[int(NR/2)], "p95", t[int(NR*0.95)],
            "max", t[NR], "ms, failed", f "/" NR }'
  wait; git rm -q --cached measure-file; rm measure-file

The skipHash tests turn split index off, because index.skipHash and a
split index don't work together at all today: with both set, "git add"
stages nothing, which Peff noticed back in 2023 [1].

[1] https://lore.kernel.org/git/20230629083859.GA585934@coredump.intra.peff.net/

Used Claude Opus 5.5, and Gemini 4 Argon for review.

Mihai Cosma (2):
  read-cache: verify skipHash index by stat data
  status: lock the index only to write it

 builtin/commit.c  |  14 +++---
 read-cache-ll.h   |   1 +
 read-cache.c      |  59 +++++++++++++++++++++++--
 repository.h      |   7 +++
 t/t7508-status.sh | 108 ++++++++++++++++++++++++++++++++++++++++++++++
 unpack-trees.c    |   1 +
 6 files changed, 178 insertions(+), 12 deletions(-)


base-commit: 47fdf2208c5b94c8cc01c106b8f7c85efb91c926
-- 
2.56.0